From 32d997e93215d7c5f99101929584e604f83e754a Mon Sep 17 00:00:00 2001 From: Malcolm Barrett Date: Mon, 14 Oct 2024 14:26:54 -0400 Subject: [PATCH 1/6] process in visual mode --- chapters/01-casual-to-causal.qmd | 32 ++-- chapters/02-whole-game.qmd | 6 +- chapters/03-counterfactuals.qmd | 102 ++++++---- chapters/04-target-trials-std-methods.qmd | 67 ++++--- chapters/05-dags.qmd | 4 +- chapters/06-not-just-a-stats-problem.qmd | 14 +- chapters/07-prep-data.qmd | 2 +- chapters/09-using-ps.qmd | 39 ++-- chapters/10-evaluating-ps.qmd | 45 +++-- chapters/11-estimands.qmd | 3 +- chapters/12-outcome-model.qmd | 64 ++++--- chapters/15-g-comp.qmd | 47 ++--- chapters/16-interaction.qmd | 1 + chapters/17-missingness-and-measurement.qmd | 6 +- chapters/18-longitudinal.qmd | 1 - chapters/21-sensitivity.qmd | 196 ++++++++++---------- chapters/22-machine-learning.qmd | 2 - chapters/24-evidence.qmd | 3 +- 18 files changed, 367 insertions(+), 267 deletions(-) diff --git a/chapters/01-casual-to-causal.qmd b/chapters/01-casual-to-causal.qmd index 0f6d46f..a6769e6 100644 --- a/chapters/01-casual-to-causal.qmd +++ b/chapters/01-casual-to-causal.qmd @@ -107,11 +107,11 @@ In 2020, particularly in the early months of the pandemic, descriptive analyses Since the coronavirus is similar to other respiratory diseases, we had many public health tools to reduce risk (e.g., distancing and, later, face masks). Descriptive statistics of cases by region were vital for deciding local policies and the strength of those policies. -A great example of a more complex descriptive analysis during the pandemic was an [ongoing report by the Financial Times of expected deaths vs. observed deaths](https://www.ft.com/content/a2901ce8-5eb7-4633-b89c-cbdf5b386938) in various countries and regions[^3]. +A great example of a more complex descriptive analysis during the pandemic was an [ongoing report by the Financial Times of expected deaths vs. observed deaths](https://www.ft.com/content/a2901ce8-5eb7-4633-b89c-cbdf5b386938) in various countries and regions[^01-casual-to-causal-1]. While the calculation of expected deaths is slightly more sophisticated than most descriptive statistics, it provided a tremendous amount of information about current deaths without needing to untangle causal effects (e.g., were they due to COVID-19 directly? Inaccessible healthcare? Cardiovascular events post-COVID?). In this (simplified) recreation of their plot from July 2020, you can see the staggering effect of the pandemic's early months. -[^3]: John Burn-Murdoch was responsible for many of these presentations and gave a [fascinating talk on the subject](https://cloud.rstudio.com/resources/rstudioglobal-2021/reporting-on-and-visualising-the-pandemic/). +[^01-casual-to-causal-1]: John Burn-Murdoch was responsible for many of these presentations and gave a [fascinating talk on the subject](https://cloud.rstudio.com/resources/rstudioglobal-2021/reporting-on-and-visualising-the-pandemic/). ```{r} #| label: fig-ft-chart @@ -213,7 +213,7 @@ It also helps us be sure that the data structure we're using matches the questio You should always do descriptive analyses of your data when conducting causal research. Finally, as we'll see in [Chapter -@sec-trials-std], there are certain circumstances where we can make causal inferences with basic statistics. -Be cautious about the distinction between the causal question and the descriptive component here, too: just because we're using the same calculation (e.g., a difference in means) doesn't mean that all descriptions you can generate are causal. +Be cautious about the distinction between the causal question and the descriptive component here, too: just because we're using the same calculation (e.g., a difference in means) doesn't mean that all descriptions you can generate are causal. Whether a descriptive analysis overlaps with a causal analysis is a function of the data and the question. ### Prediction @@ -235,7 +235,7 @@ There are many excellent texts on predictive modeling, and so we refer you to th Prediction is the most popular topic in data science, largely thanks to machine learning applications in industry. Prediction, of course, has a long history in statistics, and many models popular today have been used for decades in and outside academia. -Let's look at an example of prediction about COVID-19 [^chapter-01-1]. +Let's look at an example of prediction about COVID-19 [^01-casual-to-causal-2]. In 2021, researchers published the ISARIC 4C Deterioration model, a clinical prognostic model for predicting severe adverse outcomes for acute COVID-19 [@Gupta2021]. The authors included a descriptive analysis to understand the population from which this model was developed, particularly the distribution of the outcome and candidate predictors. One helpful aspect of this model is that it uses items commonly measured on day one of COVID-related hospitalization. @@ -244,7 +244,7 @@ The final model included eleven items and a description of their model attribute Notably, the authors used clinical domain knowledge to select candidate variables but did not fall into the temptation of interpreting the model coefficients as causal. Without question, some of the predictive value of this model stems from the causal structure of the variables as they relate to the outcome, but the researchers had a different goal entirely for this model and stuck to it. -[^chapter-01-1]: A natural model here is predicting cases, but infectious disease modeling is complex and usually uses techniques outside the usual predictive modeling workflow. +[^01-casual-to-causal-2]: A natural model here is predicting cases, but infectious disease modeling is complex and usually uses techniques outside the usual predictive modeling workflow. Here are other good examples from the predictive space: @@ -260,9 +260,9 @@ Here are other good examples from the predictive space: The key measure of validity in prediction modeling is predictive accuracy, which can be measured in several ways, such as root mean squared error (RMSE), mean absolute error (MAE), area under the curve (AUC), and many more. A convenient detail about predictive modeling is that we can often assess if we're right, which is not true of descriptive statistics for which we only have a subset of data or causal inference for which we don't know the true causal structure. -We aren't always able to assess against the truth, but it's almost always required for fitting the initial predictive model [^chapter-01-2]. +We aren't always able to assess against the truth, but it's almost always required for fitting the initial predictive model [^01-casual-to-causal-3]. -[^chapter-01-2]: We say model singular, but usually data scientists fit many models for experimentation, and often the best prediction models are some combination of predictions from several models, called a stacked model +[^01-casual-to-causal-3]: We say model singular, but usually data scientists fit many models for experimentation, and often the best prediction models are some combination of predictions from several models, called a stacked model Measurement error is also a concern for predictive modeling because we usually need accurate data for accurate predictions. Interestingly, measurement error and missingness can be informative in predictive settings. @@ -357,14 +357,14 @@ We'll come back to this topic time and time again in the book---from the basics At this point, you may wonder why the right causal model isn't just the best prediction model. It makes sense that the two would be related: naturally, things that cause other things would be predictors. -It's causality all the way down, so any predictive information *is* related, in some capacity, to the causal structure of the thing we're predicting. +It's causality all the way down, so any predictive information *is* related, in some capacity, to the causal structure of the thing we're predicting. Doesn't it stand to reason that a model that predicts well is causal, too? It's true that *some* predictive models can be great causal models and vice versa. -Unfortunately, this is not always the case; causal effects needn't predict particularly well, and good predictors needn't be causally unbiased [@shmueli2010a]. +Unfortunately, this is not always the case; causal effects needn't predict particularly well, and good predictors needn't be causally unbiased [@shmueli2010a]. There is no way to know using data alone. -Let's look at the causal perspective first because it's a bit simpler. -Consider a causally unbiased model for an exposure but only includes variables related to the outcome *and* the exposure. +Let's look at the causal perspective first because it's a bit simpler. +Consider a causally unbiased model for an exposure but only includes variables related to the outcome *and* the exposure. In other words, this model provides us with the correct answer for the exposure of interest but doesn't include other predictors of the outcome (which can sometimes be a good idea, as discussed in @sec-data-causal). If an outcome has many causes, a model that accurately describes the relationship with the exposure likely won't predict the outcome very well. Likewise, if a true causal effect of the exposure on the outcome is small, it will bring little predictive value. @@ -409,8 +409,8 @@ However, the same model in the same data with different goals will have differen ## Diagraming a causal claim {#sec-diag} -Each analysis task, whether descriptive, predictive, or inferential, should start with a clear, precise question. -Let's diagram them to understand better the structure of causal questions (to which we'll return our focus). +Each analysis task, whether descriptive, predictive, or inferential, should start with a clear, precise question. +Let's diagram them to understand better the structure of causal questions (to which we'll return our focus). Diagramming sentences is a grammatical method used to visually represent the structure of a sentence, occasionally taught in grammar school. In this technique, sentences are deconstructed into their constituent parts, such as subjects, verbs, objects, and modifiers, and then displayed using a series of lines and symbols. The arrangement of these elements on the diagram reflects their syntactic roles and how they interact within the sentence's overall structure. @@ -441,8 +441,7 @@ knitr::include_graphics("../images/sentence-diagram-2.png") ``` Let's get more specific. -A study was published in *JAMA* (the Journal of the American Medical Association) in 2005 titled "Effect of Smoking Reduction on Lung Cancer Risk." -This study concluded: "Among individuals who smoke 15 or more cigarettes per day, smoking reduction by 50% significantly reduces the risk of lung cancer". +A study was published in *JAMA* (the Journal of the American Medical Association) in 2005 titled "Effect of Smoking Reduction on Lung Cancer Risk." This study concluded: "Among individuals who smoke 15 or more cigarettes per day, smoking reduction by 50% significantly reduces the risk of lung cancer". [@godtfredsen2005effect] The study describes the time frame studied as 5-10 years. Let's diagram this causal claim. Here, we assume that the eligibility criteria and the target population for the estimated causal effect are the same (individuals who smoke 15 or more cigarettes per day); this need not always be the case. @@ -470,5 +469,4 @@ Let's return to the smoking example. Our initial question was: *Does smoking cause lung cancer?*; The evidence in the study shows: *For people who smoke 15+ cigarettes a day, reducing smoking by 50% reduces the risk of lung cancer over 5-10 years*. Does the answer match the question? Not quite. -Let's update our question to match what the study actually showed: *For people who smoke 15+ cigarettes a day, does reducing smoking by 50% reduce the lung cancer risk over 5-10 years?* -Honing this skill — asking answerable causal questions — is essential and one we will discuss throughout this book. +Let's update our question to match what the study actually showed: *For people who smoke 15+ cigarettes a day, does reducing smoking by 50% reduce the lung cancer risk over 5-10 years?* Honing this skill — asking answerable causal questions — is essential and one we will discuss throughout this book. diff --git a/chapters/02-whole-game.qmd b/chapters/02-whole-game.qmd index 769fe3b..154e619 100644 --- a/chapters/02-whole-game.qmd +++ b/chapters/02-whole-game.qmd @@ -18,7 +18,8 @@ We'll play the [whole game](https://www.gse.harvard.edu/news/uk/09/01/education- 5. Estimate the causal effect 6. Conduct sensitivity analysis on the effect estimate -We'll focus on the broader ideas behind each step and what they look like all together; however, we don't expect you to fully digest each idea. We'll spend the rest of the book taking up each step in detail. +We'll focus on the broader ideas behind each step and what they look like all together; however, we don't expect you to fully digest each idea. +We'll spend the rest of the book taking up each step in detail. ## Specify a causal question @@ -881,7 +882,8 @@ What do you think? Is this estimate reliable? Did we do a good job addressing the assumptions we need to make for a causal effect, mainly that there is no confounding? How might you criticize this model, and what would you do differently? -Ok, we know that -10 is the correct answer because the data are simulated, but in practice, we can never be sure, so we need to continue probing our assumptions until we're confident they are robust. We'll explore these techniques and others in @sec-sensitivity. +Ok, we know that -10 is the correct answer because the data are simulated, but in practice, we can never be sure, so we need to continue probing our assumptions until we're confident they are robust. +We'll explore these techniques and others in @sec-sensitivity. To calculate this effect, we: diff --git a/chapters/03-counterfactuals.qmd b/chapters/03-counterfactuals.qmd index dc601a9..6c0d3e9 100644 --- a/chapters/03-counterfactuals.qmd +++ b/chapters/03-counterfactuals.qmd @@ -112,7 +112,8 @@ In any case, we must rely on statistical techniques to help construct these unob Let's suppose some happiness index, from 1-10 exists. We are interested in assessing whether eating chocolate ice cream versus vanilla will increase happiness. -We have 10 individuals with two potential outcomes for each, one is what their happiness would be if they ate chocolate ice cream, (defined as `y_chocolate` in the code below), and one is what their happiness would be if they ate vanilla ice cream (defined as `y_vanilla` in the code below). We can define the true causal effect of eating chocolate ice cream (versus vanilla) on happiness for each individual as the difference between the two (@tbl-po). +We have 10 individuals with two potential outcomes for each, one is what their happiness would be if they ate chocolate ice cream, (defined as `y_chocolate` in the code below), and one is what their happiness would be if they ate vanilla ice cream (defined as `y_vanilla` in the code below). +We can define the true causal effect of eating chocolate ice cream (versus vanilla) on happiness for each individual as the difference between the two (@tbl-po). ```{r} #| label: sim-generation @@ -147,7 +148,6 @@ data <- data |> data ``` - ```{r} #| label: tbl-po #| tbl-cap: "Potential Outcomes Simulation: The causal effect of eating chocolate (versus vanilla) ice cream on happiness" @@ -168,9 +168,12 @@ data |> ) ``` -For example, examining @tbl-po, the causal effect of eating chocolate ice cream (versus vanilla) for individual `4` is `r data |> filter(id == 4) |> pull(causal_effect)`, whereas the causal effect for individual `9` is `r data |> filter(id == 9) |> pull(causal_effect)`. The *average* potential happiness after eating chocolate is `r mean(data |> pull(y_chocolate))` and the *average* potential happiness after eating vanilla is `r mean(data |> pull(y_vanilla))`. The *average* treatment effect of eating chocolate (versus vanilla) ice cream among the ten individuals in this study is `r mean(data |> pull(causal_effect))`. +For example, examining @tbl-po, the causal effect of eating chocolate ice cream (versus vanilla) for individual `4` is `r data |> filter(id == 4) |> pull(causal_effect)`, whereas the causal effect for individual `9` is `r data |> filter(id == 9) |> pull(causal_effect)`. +The *average* potential happiness after eating chocolate is `r mean(data |> pull(y_chocolate))` and the *average* potential happiness after eating vanilla is `r mean(data |> pull(y_vanilla))`. +The *average* treatment effect of eating chocolate (versus vanilla) ice cream among the ten individuals in this study is `r mean(data |> pull(causal_effect))`. -In reality, we cannot observe both potential outcomes, in any moment in time, each individual in our study can only eat *one* flavor of ice cream. Suppose we let our participants choose which ice cream they wanted to eat and each choose their favorite (i.e. they knew which would make them "happier" and picked that one. Now what we *observe* is shown in @tbl-obs. +In reality, we cannot observe both potential outcomes, in any moment in time, each individual in our study can only eat *one* flavor of ice cream. +Suppose we let our participants choose which ice cream they wanted to eat and each choose their favorite (i.e. they knew which would make them "happier" and picked that one. Now what we *observe* is shown in @tbl-obs. ```{r} #| results: hide @@ -192,7 +195,6 @@ data_observed <- data |> data_observed ``` - ```{r} #| label: tbl-obs #| tbl-cap: "Potential Outcomes Simulation: The observed exposure and outcome used to estimate the effect of eating chocolate (versus vanilla) ice cream on happiness" @@ -220,9 +222,13 @@ data_observed |> summarise(avg_outcome = mean(observed_outcome)) ``` -Now, the *observed* average outcome among those who ate chocolate ice cream is `r round(avg_chocolate, 1)` (the same as the true average potential outcome), while the *observed* average outcome among those who ate vanilla is `r round(avg_vanilla, 1)` -- quite different from the *actual* average (`r mean(data |> pull(y_vanilla))`). The estimated causal effect here could be calculated as `r round(avg_chocolate, 1)` - `r round(avg_vanilla, 1)` = `r round(avg_chocolate - avg_vanilla, 1)`. +Now, the *observed* average outcome among those who ate chocolate ice cream is `r round(avg_chocolate, 1)` (the same as the true average potential outcome), while the *observed* average outcome among those who ate vanilla is `r round(avg_vanilla, 1)` -- quite different from the *actual* average (`r mean(data |> pull(y_vanilla))`). +The estimated causal effect here could be calculated as `r round(avg_chocolate, 1)` - `r round(avg_vanilla, 1)` = `r round(avg_chocolate - avg_vanilla, 1)`. -It turns out here, these 10 participants *chose* which ice cream they wanted to eat and they always chose to eat their favorite! This artificially made it look like eating vanilla ice cream would increase the happiness in this population when in fact we know the opposite is true. The next section will discuss which assumptions need to be true in order to allow us to *accurately* estimate causal effects using observed data. As a sneak peak, our issue here was that how the exposure was decided, if instead we *randomized* who ate chocolate versus vanilla ice cream we would (on average, with a large enough sample) recover the true causal effect. +It turns out here, these 10 participants *chose* which ice cream they wanted to eat and they always chose to eat their favorite! +This artificially made it look like eating vanilla ice cream would increase the happiness in this population when in fact we know the opposite is true. +The next section will discuss which assumptions need to be true in order to allow us to *accurately* estimate causal effects using observed data. +As a sneak peak, our issue here was that how the exposure was decided, if instead we *randomized* who ate chocolate versus vanilla ice cream we would (on average, with a large enough sample) recover the true causal effect. ```{r} ## we are doing something *random* so let's set a seed so we always observe the @@ -257,11 +263,10 @@ At any given time, only one of these *potential outcomes* is observable -- namel Under certain assumptions, we can leverage data from individuals exposed to different inputs to compare the average differences in their observed outcomes. The most common assumptions across the approaches we describe in this book are: -1. **Consistency**: We assume that the causal question you claim you are answering is consistent with the one you are *actually* answering with your analysis. Mathematically, this means that $Y_{obs} = (X)Y(1) + (1 - X)Y(0)$, in other words, the outcome you observe is exactly equal to the potential outcome under the exposure you received. Two common ways to discuss this assumption are: - * **Well defined exposure**: We assume that for each value of the exposure, there is no difference between subjects in the delivery of that exposure. -Put another way, multiple versions of the treatment do not exist. - * **No interference**: We assume that the outcome (technically all *potential* outcomes, regardless of whether they are observed) for any subject does not depend on another subject's exposure. - +1. **Consistency**: We assume that the causal question you claim you are answering is consistent with the one you are *actually* answering with your analysis. Mathematically, this means that $Y_{obs} = (X)Y(1) + (1 - X)Y(0)$, in other words, the outcome you observe is exactly equal to the potential outcome under the exposure you received. Two common ways to discuss this assumption are: + - **Well defined exposure**: We assume that for each value of the exposure, there is no difference between subjects in the delivery of that exposure. Put another way, multiple versions of the treatment do not exist. + - **No interference**: We assume that the outcome (technically all *potential* outcomes, regardless of whether they are observed) for any subject does not depend on another subject's exposure. + ::: callout-tip ## Jargon @@ -270,42 +275,46 @@ Likewise, these assumptions are sometimes referred to as *identifiability condit ::: 2. **Exchangeability**: We assume that within levels of relevant variables (confounders), exposed and unexposed subjects have an equal likelihood of experiencing any outcome prior to exposure; i.e. the exposed and unexposed subjects are exchangeable. -This assumption is sometimes referred to as **no unmeasured confounding**. + This assumption is sometimes referred to as **no unmeasured confounding**. 3. **Positivity**: We assume that within each level and combination of the study variables used to achieve exchangeability, there are exposed and unexposed subjects. -Said differently, each individual has some chance of experiencing every available exposure level. -Sometimes this is referred to as the **probabilistic** assumption. - - - + Said differently, each individual has some chance of experiencing every available exposure level. + Sometimes this is referred to as the **probabilistic** assumption. ::: callout-note ## Apples-to-apples -Practically, most of the assumptions we need to make for causal inference are so we can make an *apples-to-apples* comparison: we want to make sure we're comparing individuals that are similar --- who would serve as good proxies for each other's counterfactuals. +Practically, most of the assumptions we need to make for causal inference are so we can make an *apples-to-apples* comparison: we want to make sure we're comparing individuals that are similar --- who would serve as good proxies for each other's counterfactuals. -The phrase *apples-to-apples* stems from the saying "comparing apples to oranges", e.g. comparing two things that are incomparable. +The phrase *apples-to-apples* stems from the saying "comparing apples to oranges", e.g. comparing two things that are incomparable. -That's only one way to say it. [There are a lot of variations worldwide](https://en.wikipedia.org/wiki/Apples_and_oranges). Here are some other things people incorrectly compare: +That's only one way to say it. +[There are a lot of variations worldwide](https://en.wikipedia.org/wiki/Apples_and_oranges). +Here are some other things people incorrectly compare: -* Cheese and chalk (UK English) -* Apples and pears (German) -* Potatoes and sweet potatoes (Latin American Spanish) -* Grandmothers and toads (Serbian) -* Horses and donkeys (Hindi) +- Cheese and chalk (UK English) +- Apples and pears (German) +- Potatoes and sweet potatoes (Latin American Spanish) +- Grandmothers and toads (Serbian) +- Horses and donkeys (Hindi) ::: ### Causal Assumptions Simulation -Let's bring back our simulation from @sec-po-sim. Recall that we have individuals who will either eat chocolate or vanilla ice cream and we are interested in assessing the causal effect of this exposure on their happiness. Let's see how violations of each assumption may impact the estimation of the causal effect. +Let's bring back our simulation from @sec-po-sim. +Recall that we have individuals who will either eat chocolate or vanilla ice cream and we are interested in assessing the causal effect of this exposure on their happiness. +Let's see how violations of each assumption may impact the estimation of the causal effect. #### Consistency violation -Two ways the consistency assumption can be violated is (1) lack of a well defined exposure and (2) interference. Let's see how these impact our ability to accurately estimate a causal effect. +Two ways the consistency assumption can be violated is (1) lack of a well defined exposure and (2) interference. +Let's see how these impact our ability to accurately estimate a causal effect. ##### Well defined exposure -Suppose that there were in fact two containers of chocolate ice cream, one of which was spoiled. Therefore, despite the fact that having an exposure "chocolate" could mean different things depending on where the individual's scoop came from (regular chocolate ice cream, or spoiled chocolate ice cream), we are lumping them all together under a single umbrella (hence the violation, we have "multiple versions of treatment"). You can see how this falls under consistency because the issue here is that the potential outcome we think we are estimating is not the one we are actually observing. +Suppose that there were in fact two containers of chocolate ice cream, one of which was spoiled. +Therefore, despite the fact that having an exposure "chocolate" could mean different things depending on where the individual's scoop came from (regular chocolate ice cream, or spoiled chocolate ice cream), we are lumping them all together under a single umbrella (hence the violation, we have "multiple versions of treatment"). +You can see how this falls under consistency because the issue here is that the potential outcome we think we are estimating is not the one we are actually observing. ```{r} data <- data.frame( @@ -341,11 +350,14 @@ data_observed |> summarise(avg_outcome = mean(observed_outcome)) ``` -We know the *true* average causal effect of (unspoiled) chocolate in the sample is `r mean(data |> pull(causal_effect))`, however our estimated causal effect (because our data are not consistent with the question we are asking) is `r data_observed |> group_by(exposure) |> summarise(avg_outcome = mean(observed_outcome)) |> mutate(exposure = c(1, 0)) |> pivot_wider(names_from = exposure, values_from = avg_outcome, names_prefix = "x_") |> summarise(estimate = x_1 - x_0) |> pull() |> round(1)`. This demonstrates what can go wrong when *well defined exposure* is violated. +We know the *true* average causal effect of (unspoiled) chocolate in the sample is `r mean(data |> pull(causal_effect))`, however our estimated causal effect (because our data are not consistent with the question we are asking) is `r data_observed |> group_by(exposure) |> summarise(avg_outcome = mean(observed_outcome)) |> mutate(exposure = c(1, 0)) |> pivot_wider(names_from = exposure, values_from = avg_outcome, names_prefix = "x_") |> summarise(estimate = x_1 - x_0) |> pull() |> round(1)`. +This demonstrates what can go wrong when *well defined exposure* is violated. -##### Interference +##### Interference -Interference would mean that an individual's exposure impacts another's potential outcome. For example, let's say each individual has a partner, and their potential outcome depends on both what flavor of ice cream they ate *and* what flavor their partner ate. For example, in the simulation below, having a partner that received a *different* flavor of ice cream increases their happiness by two units. +Interference would mean that an individual's exposure impacts another's potential outcome. +For example, let's say each individual has a partner, and their potential outcome depends on both what flavor of ice cream they ate *and* what flavor their partner ate. +For example, in the simulation below, having a partner that received a *different* flavor of ice cream increases their happiness by two units. ```{r} data <- data.frame( @@ -383,7 +395,15 @@ data_observed |> group_by(exposure) |> summarise(avg_outcome = mean(observed_outcome)) ``` -Now our estimated causal effect (because interference exists) is `r data_observed |> group_by(exposure) |> summarise(avg_outcome = mean(observed_outcome)) |> mutate(exposure = c(1, 0)) |> pivot_wider(names_from = exposure, values_from = avg_outcome, names_prefix = "x_") |> summarise(estimate = x_1 - x_0) |> pull() |> round(1)`. This demonstrates what can go wrong when *interference* occurs. One of the main ways to combat interference is change the *unit* under consideration. Here, each individual, each unique *id*, is considered a unit, and there is interference between units (i.e. between partners). If instead we consider each *partner* as a unit and randomize the partners rather than the individuals, we solve the interference issue, as there is not interference *between* different partner sets. This is sometimes referred to as a *cluster randomized trial*. What we decide to do within each cluster may depend on the causal question at hand. For example, if we want to know what would happen if *everyone* ate chocolate ice cream versus if *everyone* ate vanilla, we would want to randomize both partners to either chocolate or vanilla, as seen below. + +Now our estimated causal effect (because interference exists) is `r data_observed |> group_by(exposure) |> summarise(avg_outcome = mean(observed_outcome)) |> mutate(exposure = c(1, 0)) |> pivot_wider(names_from = exposure, values_from = avg_outcome, names_prefix = "x_") |> summarise(estimate = x_1 - x_0) |> pull() |> round(1)`. +This demonstrates what can go wrong when *interference* occurs. +One of the main ways to combat interference is change the *unit* under consideration. +Here, each individual, each unique *id*, is considered a unit, and there is interference between units (i.e. between partners). +If instead we consider each *partner* as a unit and randomize the partners rather than the individuals, we solve the interference issue, as there is not interference *between* different partner sets. +This is sometimes referred to as a *cluster randomized trial*. +What we decide to do within each cluster may depend on the causal question at hand. +For example, if we want to know what would happen if *everyone* ate chocolate ice cream versus if *everyone* ate vanilla, we would want to randomize both partners to either chocolate or vanilla, as seen below. ```{r} set.seed(11) @@ -415,7 +435,8 @@ data_observed |> #### Exchangeability violation -We have actually already seen an example of an exchangeability violation in @sec-po-sim. In that example, participants were able to choose the ice cream that they wanted to eat, so people who were more likely to have a positive effect from eating chocolate chose that, and those more likely to have a positive effect from eating vanilla chose that. +We have actually already seen an example of an exchangeability violation in @sec-po-sim. +In that example, participants were able to choose the ice cream that they wanted to eat, so people who were more likely to have a positive effect from eating chocolate chose that, and those more likely to have a positive effect from eating vanilla chose that. ```{r} data <- data.frame( @@ -443,11 +464,16 @@ data_observed |> summarise(avg_outcome = mean(observed_outcome)) ``` -How could we correct this? If we had some people who preferred chocolate ice cream but ended up taking vanilla instead, we could *adjust* for the preference, and the effect conditioned on this would no longer have an exchangeability issue. It turns out that this example as we have constructed it doesn't lend itself to this solution because participants chose their preferred flavor 100% of the time making this *also* a positivity violation. +How could we correct this? +If we had some people who preferred chocolate ice cream but ended up taking vanilla instead, we could *adjust* for the preference, and the effect conditioned on this would no longer have an exchangeability issue. +It turns out that this example as we have constructed it doesn't lend itself to this solution because participants chose their preferred flavor 100% of the time making this *also* a positivity violation. #### Positivity violation -As stated above, the previous example violates both *exchangeability* and *positivity*. How could we fix it? As long as *some* people chose outside their preference with some probability (even if it is small!) we can remove this violation. Let's say instead of everyone picking their flavor of preference 100% of the time, they just had a 80% chance of picking that flavor. +As stated above, the previous example violates both *exchangeability* and *positivity*. +How could we fix it? +As long as *some* people chose outside their preference with some probability (even if it is small!) we can remove this violation. +Let's say instead of everyone picking their flavor of preference 100% of the time, they just had a 80% chance of picking that flavor. ```{r} data <- data.frame( @@ -478,7 +504,9 @@ lm( data_observed ) ``` -After *adjusting* for this variable (chocolate preference), we recover the correct causal effect. This value is not exactly the same as the truth we obtain with the (unobservable) potential outcomes because we are dealing with a small sample -- as our sample size increases this will get closer to the truth. + +After *adjusting* for this variable (chocolate preference), we recover the correct causal effect. +This value is not exactly the same as the truth we obtain with the (unobservable) potential outcomes because we are dealing with a small sample -- as our sample size increases this will get closer to the truth. Causal assumptions can be difficult to verify and may not hold for many data collection strategies. We cannot overstate the importance of checking these criteria to the extent possible! diff --git a/chapters/04-target-trials-std-methods.qmd b/chapters/04-target-trials-std-methods.qmd index dee2069..a0b0f93 100644 --- a/chapters/04-target-trials-std-methods.qmd +++ b/chapters/04-target-trials-std-methods.qmd @@ -26,35 +26,38 @@ In reality, we often see this assumption violated by issues such as *drop out* o If there is differential drop out between exposure groups (for example, if participants randomly assigned to the treatment are more likely to drop out of a study, and thus we don't observe their outcome), then the observed exposure groups are no longer *exchangeable*. Therefore, in @tbl-assump-solved we have two columns, one for the *ideal* randomized trial (where adherence is assumed to be perfect and no participants drop out) and one for *realistic* randomized trials where this may not be so. -| Assumption | Ideal Randomized Trial | Realistic Randomized Trial | Observational Study | -|-----------------|-----------------|---------------------|-------------------| +| Assumption | Ideal Randomized Trial | Realistic Randomized Trial | Observational Study | +|-----------------|-----------------|--------------------|------------------| | Consistency (Well defined exposure) | `r emo::ji("smile")` | `r emo::ji("smile")` | `r emo::ji("shrug")` | | Consistency (No interference) | `r emo::ji("shrug")` | `r emo::ji("shrug")` | `r emo::ji("shrug")` | -| Positivity | `r emo::ji("smile")` | `r emo::ji("smile")` | `r emo::ji("shrug")` | -| Exchangeability | `r emo::ji("smile")` | `r emo::ji("shrug")` | `r emo::ji("shrug")` | +| Positivity | `r emo::ji("smile")` | `r emo::ji("smile")` | `r emo::ji("shrug")` | +| Exchangeability | `r emo::ji("smile")` | `r emo::ji("shrug")` | `r emo::ji("shrug")` | : Assumptions solved by study design. `r emo::ji("smile")` indicates it is solved by default, `r emo::ji("shrug")` indicates that it is *solvable* but not solved by default. {#tbl-assump-solved} -When designing a study, the first step is asking an appropriate *causal question*. We then can map this question to a *protocol*, consisting of the following seven elements, as defined by @hernan2016using: +When designing a study, the first step is asking an appropriate *causal question*. +We then can map this question to a *protocol*, consisting of the following seven elements, as defined by @hernan2016using: -* Eligibility criteria -* Exposure definition -* Assignment procedures -* Follow-up period -* Outcome definition -* Causal contrast of interest -* Analysis plan +- Eligibility criteria +- Exposure definition +- Assignment procedures +- Follow-up period +- Outcome definition +- Causal contrast of interest +- Analysis plan -In @tbl-protocol we map each of these elements to the corresponding assumption that it can address. For example, exchangeability can be addressed by the eligibility criteria (we can restrict our study to only participants for whom exposure assignment is exchangeable), assignment procedure (we could use random exposure assignment to ensure exchangeability), follow-up period (we can be sure to choose an appropriate start time for our follow-up period to ensure that we are not inducing bias -- we'll think more about this in a future chapter), and/or the analysis plan (we can adjust for any factors that would cause those in different exposure groups to lack exchangeability). +In @tbl-protocol we map each of these elements to the corresponding assumption that it can address. +For example, exchangeability can be addressed by the eligibility criteria (we can restrict our study to only participants for whom exposure assignment is exchangeable), assignment procedure (we could use random exposure assignment to ensure exchangeability), follow-up period (we can be sure to choose an appropriate start time for our follow-up period to ensure that we are not inducing bias -- we'll think more about this in a future chapter), and/or the analysis plan (we can adjust for any factors that would cause those in different exposure groups to lack exchangeability). -Assumption | Eligibility Criteria | Exposure Definition| Assignment Procedures | Follow-up Period | Outcome Definition | Causal contrast | Analysis Plan -------------|----------------- | ------------------|---------|----------|----------|--------- | ------- -Consistency (Well defined exposure) |`r emo::ji("heavy_check_mark")`|`r emo::ji("heavy_check_mark")`|| | | -Consistency (No interference) | | `r emo::ji("heavy_check_mark")`|`r emo::ji("heavy_check_mark")` | | `r emo::ji("heavy_check_mark")` | | `r emo::ji("heavy_check_mark")` -Positivity |`r emo::ji("heavy_check_mark")`||`r emo::ji("heavy_check_mark")`|| | | `r emo::ji("heavy_check_mark")` -Exchangeability |`r emo::ji("heavy_check_mark")`||`r emo::ji("heavy_check_mark")`|`r emo::ji("heavy_check_mark")`|| | `r emo::ji("heavy_check_mark")` +| Assumption | Eligibility Criteria | Exposure Definition | Assignment Procedures | Follow-up Period | Outcome Definition | Causal contrast | Analysis Plan | +|---------|---------|---------|---------|---------|---------|---------|---------| +| Consistency (Well defined exposure) | `r emo::ji("heavy_check_mark")` | `r emo::ji("heavy_check_mark")` | | | | | | +| Consistency (No interference) | | `r emo::ji("heavy_check_mark")` | `r emo::ji("heavy_check_mark")` | | `r emo::ji("heavy_check_mark")` | | `r emo::ji("heavy_check_mark")` | +| Positivity | `r emo::ji("heavy_check_mark")` | | `r emo::ji("heavy_check_mark")` | | | | `r emo::ji("heavy_check_mark")` | +| Exchangeability | `r emo::ji("heavy_check_mark")` | | `r emo::ji("heavy_check_mark")` | `r emo::ji("heavy_check_mark")` | | | `r emo::ji("heavy_check_mark")` | + : Mapping assumptions to elements of a study protocol {#tbl-protocol} Recall our diagrams from @sec-diag (@fig-diagram-4); several of these protocol elements can be mapped to these diagrams when we are attempting to define our causal question. @@ -86,16 +89,17 @@ ggplot(data, aes(x = x, y = y)) + theme_void() ``` - ## Target Trials -There are many reasons why randomization may not be possible. For example, it might not be ethical to randomly assign people to a particular exposure, there may not be funding available to run a randomized trial, or there might not be enough time to conduct a full trial. In these situations, we rely on observational data to help us answer causal questions by implementing a *target trial*. +There are many reasons why randomization may not be possible. +For example, it might not be ethical to randomly assign people to a particular exposure, there may not be funding available to run a randomized trial, or there might not be enough time to conduct a full trial. +In these situations, we rely on observational data to help us answer causal questions by implementing a *target trial*. A *target trial* answers: What experiment would you design if you could? Specifying a target trial is nearly identical to the process we described for a randomized trial. -We define eligibility, exposure, follow-up period, outcome, estimate of interest, and the analysis plan. +We define eligibility, exposure, follow-up period, outcome, estimate of interest, and the analysis plan. The key difference with the target trial in the observational setting, of course, is that we cannot assign exposure. -The analysis planning and execution step of the target trial is the most technically involved and a core focus of this book; e.g. using DAGs to ensure that we have measured and are controlling for the right set of confounders, composing statistical programs that invoke an appropriate adjustment method such as IP weighting, and conducting sensitivity analyses to assess how sensitive our conclusions are to unmeasured confounding or misspecification. +The analysis planning and execution step of the target trial is the most technically involved and a core focus of this book; e.g. using DAGs to ensure that we have measured and are controlling for the right set of confounders, composing statistical programs that invoke an appropriate adjustment method such as IP weighting, and conducting sensitivity analyses to assess how sensitive our conclusions are to unmeasured confounding or misspecification. ## Causal inference with `group_by()` and `summarize()` {#sec-group-sum} @@ -179,9 +183,13 @@ sim |> select(confounder, exposure, outcome, observed_potential_outcome) ``` -Great! Let's begin by proving to ourselves that this violates the exchangeability assumption. Recall from @sec-assump: +Great! +Let's begin by proving to ourselves that this violates the exchangeability assumption. +Recall from @sec-assump: -> **Exchangeability**: We assume that within levels of relevant variables (confounders), exposed and unexposed subjects have an equal likelihood of experiencing any outcome prior to exposure; i.e. the exposed and unexposed subjects are exchangeable. This assumption is sometimes referred to as **no unmeasured confounding**, though exchangeability implies more than that, such as no selection bias and that confounder relationships are appropriately specified. We will further define exchangeability through the lens of DAGs in the next chapter. +> **Exchangeability**: We assume that within levels of relevant variables (confounders), exposed and unexposed subjects have an equal likelihood of experiencing any outcome prior to exposure; i.e. the exposed and unexposed subjects are exchangeable. +> This assumption is sometimes referred to as **no unmeasured confounding**, though exchangeability implies more than that, such as no selection bias and that confounder relationships are appropriately specified. +> We will further define exchangeability through the lens of DAGs in the next chapter. Now, let's try to estimate the effect of the `exposure` on the `outcome` assuming the two exposed groups are exchangeable. @@ -260,7 +268,10 @@ Great! Now our estimate is much closer to the true value (0). ::: callout -The method we are using to solve the fact that our two groups are not exchangeable is known as **stratification**. We are *stratifying* by the confounder(s) and estimating the causal effect within each stratum. To get an overall average causal effect we are averaging across the strata. This can be a great tool if there are very few confounders, however it can suffer from the curse of dimensionality as the number of confounders as well as the number of levels within each confounder increases. +The method we are using to solve the fact that our two groups are not exchangeable is known as **stratification**. +We are *stratifying* by the confounder(s) and estimating the causal effect within each stratum. +To get an overall average causal effect we are averaging across the strata. +This can be a great tool if there are very few confounders, however it can suffer from the curse of dimensionality as the number of confounders as well as the number of levels within each confounder increases. ::: ### Two binary confounders @@ -436,7 +447,6 @@ dagify( Let's examine three models: (1) an unadjusted model (@tbl-panel-1), (2) a linear model that adjusts for the baseline covariates (@tbl-panel-2), and (3) a propensity score weighted model (@tbl-panel-3). - ```{r} #| label: tbl-panel #| layout-ncol: 2 @@ -501,7 +511,8 @@ Standard methods can still estimate unbiased effects, but more care needs to be For example, we need the exposed an unexposed groups to be *exchangeable*; this means we must adjust for *all confounders* with their correct functional form. If everything is simple and linear (and there is no effect heterogeneity, that is everyone's causal effect is the same regardless of their baseline factors), then a simple regression model that adjusts for the confounders can give you an unbiased result. Let's look at a simple example such as this. -Notice in the simulation below, the main difference compared to the above simulation is that the probability of treatment assignment is no longer 0.5 as it was above, but now dependent on the participants `age` and `weight`. For example, maybe doctors tend to prescribe a certain treatment to patients who are older and who weigh more. +Notice in the simulation below, the main difference compared to the above simulation is that the probability of treatment assignment is no longer 0.5 as it was above, but now dependent on the participants `age` and `weight`. +For example, maybe doctors tend to prescribe a certain treatment to patients who are older and who weigh more. The true causal effect is still 1, but now we have two confounders, `age` and `weight` (@fig-diag-2). ```{r} diff --git a/chapters/05-dags.qmd b/chapters/05-dags.qmd index d6836d1..4767f1c 100644 --- a/chapters/05-dags.qmd +++ b/chapters/05-dags.qmd @@ -64,13 +64,13 @@ dag_data |> ) ``` -The type of causal diagrams we use are also called directed acyclic graphs (DAGs)[^1]. +The type of causal diagrams we use are also called directed acyclic graphs (DAGs)[^05-dags-1]. These graphs are directed because they include arrows going in a specific direction. They're acyclic because they don't go in circles; a variable can't cause itself, for instance. DAGs are used for various problems, but we're specifically concerned with *causal* DAGs. This class of DAGs is sometimes called Structural Causal Models (SCMs) because they are a model of the causal structure of a question [@hernan2021; @Pearl_Glymour_Jewell_2021]. -[^1]: An essential but rarely observed detail of DAGs is that dag is also an [affectionate Australian insult](https://en.wikipedia.org/wiki/Dag_(slang)) referring to the dung-caked fur of a sheep, a *daglock*. +[^05-dags-1]: An essential but rarely observed detail of DAGs is that dag is also an [affectionate Australian insult](https://en.wikipedia.org/wiki/Dag_(slang)) referring to the dung-caked fur of a sheep, a *daglock*. DAGs depict causal relationships between variables. Visually, the way they depict variables is as *edges* and *nodes*. diff --git a/chapters/06-not-just-a-stats-problem.qmd b/chapters/06-not-just-a-stats-problem.qmd index c447727..21ba29d 100644 --- a/chapters/06-not-just-a-stats-problem.qmd +++ b/chapters/06-not-just-a-stats-problem.qmd @@ -164,12 +164,17 @@ causal_quartet |> facet_wrap(~dataset) ``` -::: {.callout-tip} +::: callout-tip ## Why did we standardize the coefficients? -Standardizing numeric variables to have a mean of 0 and standard deviation of 1, as implemented in `scale()`, is a common technique in statistics. It's useful for a variety of reasons, but we chose to scale the variables here to emphasize the identical correlation between `covariate` and `exposure` in each dataset. If we didn't scale the variables, the correlation would be the same, but the plots would look different because their standard deviation are different. The beta coefficient in an OLS model is calculated with information about the covariance and the standard deviation of the variable, so scaling it makes the coefficient identical to the Pearson's correlation. +Standardizing numeric variables to have a mean of 0 and standard deviation of 1, as implemented in `scale()`, is a common technique in statistics. +It's useful for a variety of reasons, but we chose to scale the variables here to emphasize the identical correlation between `covariate` and `exposure` in each dataset. +If we didn't scale the variables, the correlation would be the same, but the plots would look different because their standard deviation are different. +The beta coefficient in an OLS model is calculated with information about the covariance and the standard deviation of the variable, so scaling it makes the coefficient identical to the Pearson's correlation. -@fig-causal_quartet_covariate_unscaled shows the unscaled relationship between `covariate` and `exposure`. Now, we see some differences: dataset 4 seems to have more variance in `covariate`, but that's not actionable information. In fact, it's a mathematical artifact of the data generating process. +@fig-causal_quartet_covariate_unscaled shows the unscaled relationship between `covariate` and `exposure`. +Now, we see some differences: dataset 4 seems to have more variance in `covariate`, but that's not actionable information. +In fact, it's a mathematical artifact of the data generating process. ```{r} #| label: fig-causal_quartet_covariate_unscaled @@ -631,7 +636,8 @@ But look again. `exposure` is a mediator for `covariate`'s effect on `outcome`; some of the total effect is mediated through `outcome`, while there is also a direct effect of `covariate` on `outcome`. **Both estimates are unbiased, but they are different *types* of estimates**. The effect of `exposure` on `outcome` is the *total effect* of that relationship, while the effect of `covariate` on `outcome` is the *direct effect*. [^06-not-just-a-stats-problem-4]: Additionally, OLS produces a *collapsible* effect. - Other effects, like the odds and hazards ratios, are *non-collapsible*, meaning that the conditional odds or hazards ratio might differ from its marginal version, even when there is no confounding. We'll discuss non-collapsibility in @sec-non-collapse. + Other effects, like the odds and hazards ratios, are *non-collapsible*, meaning that the conditional odds or hazards ratio might differ from its marginal version, even when there is no confounding. + We'll discuss non-collapsibility in @sec-non-collapse. ```{r} #| label: fig-quartet_confounder diff --git a/chapters/07-prep-data.qmd b/chapters/07-prep-data.qmd index e31c94b..965b78f 100644 --- a/chapters/07-prep-data.qmd +++ b/chapters/07-prep-data.qmd @@ -147,7 +147,7 @@ Most of our data manipulation tools come from the `{dplyr}` package (@tbl-dplyr) We will also use `{lubridate}` to help us manipulate dates. | Target trial protocol element | {dplyr} functions | -|-------------------------------|---------------------------------------------| +|------------------------------|------------------------------------------| | Eligibility criteria | `filter()` | | Exposure definition | `mutate()` | | Assignment procedures | `mutate()` | diff --git a/chapters/09-using-ps.qmd b/chapters/09-using-ps.qmd index d3c3bf1..af74ad8 100644 --- a/chapters/09-using-ps.qmd +++ b/chapters/09-using-ps.qmd @@ -8,12 +8,19 @@ status("polishing") ``` - -The propensity score is a *balancing* tool -- we use it to help us make our exposure groups *exchangeable*. There are many ways to incorporate the propensity score into an analysis. Commonly used techniques include stratification (estimating the causal effect within propensity score stratum), matching, weighting, and direct covariate adjustment. In this section, we will focus on *matching* and *weighting*; other techniques will be discussed once we introduce the *outcome model*. Recall at this point in the book we are still in the *design* phase. We have not yet incorporated the outcome into our analysis at all. +The propensity score is a *balancing* tool -- we use it to help us make our exposure groups *exchangeable*. +There are many ways to incorporate the propensity score into an analysis. +Commonly used techniques include stratification (estimating the causal effect within propensity score stratum), matching, weighting, and direct covariate adjustment. +In this section, we will focus on *matching* and *weighting*; other techniques will be discussed once we introduce the *outcome model*. +Recall at this point in the book we are still in the *design* phase. +We have not yet incorporated the outcome into our analysis at all. ## Matching -Ultimately, we want the exposed and unexposed observations to be *exchangeable* with respect to the confounders we have proposed in our DAG (so we can use the observed effect for one to estimate the counterfactual for the other). One way to do this is to ensure that each observation in our analysis sample has at least one observation of the opposite exposure that has *match*ing values for each of these confounders. If we had a small number of binary confounders, for example, we might be able to construct an *exact match* for observations (and only include those for whom such a match exists), but as the number and continuity of confounders increases, exact matching becomes less feasible. This is where the propensity score, a summary measure of all of the confounders, comes in to play. +Ultimately, we want the exposed and unexposed observations to be *exchangeable* with respect to the confounders we have proposed in our DAG (so we can use the observed effect for one to estimate the counterfactual for the other). +One way to do this is to ensure that each observation in our analysis sample has at least one observation of the opposite exposure that has *match*ing values for each of these confounders. +If we had a small number of binary confounders, for example, we might be able to construct an *exact match* for observations (and only include those for whom such a match exists), but as the number and continuity of confounders increases, exact matching becomes less feasible. +This is where the propensity score, a summary measure of all of the confounders, comes in to play. Let's setup the data as we did in @sec-building-models. @@ -24,7 +31,11 @@ library(touringplans) seven_dwarfs_9 <- seven_dwarfs_train_2018 |> filter(wait_hour == 9) ``` -We can re-fit the propensity score using the `{MatchIt}` package, as below. Notice here the `matchit` function fit a logistic regression model for our propensity score, as we had in @sec-building-models. There were 60 days in 2018 where the Magic Kingdom had extra magic morning hours. For each of these 60 exposed days, `matchit` found a comparable unexposed day, by implementing a nearest-neighbor match using the constructed propensity score. Examining the output, we also see that the target estimand is an "ATT" (do not worry about this yet, we will discuss this and several other estimands in @sec-estimands). +We can re-fit the propensity score using the `{MatchIt}` package, as below. +Notice here the `matchit` function fit a logistic regression model for our propensity score, as we had in @sec-building-models. +There were 60 days in 2018 where the Magic Kingdom had extra magic morning hours. +For each of these 60 exposed days, `matchit` found a comparable unexposed day, by implementing a nearest-neighbor match using the constructed propensity score. +Examining the output, we also see that the target estimand is an "ATT" (do not worry about this yet, we will discuss this and several other estimands in @sec-estimands). ```{r} library(MatchIt) @@ -34,7 +45,9 @@ m <- matchit( ) m ``` -We can use the `get_matches` function to create a data frame with the original variables that only consists of those who were matched. Notice here our sample size has been reduced from the original 354 days to 120. + +We can use the `get_matches` function to create a data frame with the original variables that only consists of those who were matched. +Notice here our sample size has been reduced from the original 354 days to 120. ```{r} matched_data <- get_matches(m) @@ -43,11 +56,19 @@ glimpse(matched_data) ## Weighting -One way to think about matching is as a crude "weight" where everyone who was matched gets a weight of 1 and everyone who was not matched gets a weight of 0 in the final sample. Another option is to allow this weight to be smooth, applying a weight to allow, on average, the covariates of interest to be balanced in the weighted population. To do this, we will construct a weight using the propensity score. There are many different weights that can be applied, depending on your target estimand of interest (see @sec-estimands for details). For this section, we will focus on the "Average Treatment Effect" weights, commonly referred to as an "inverse probability weight". The weight is constructed as follows, where each observation is weighted by the *inverse* of the probability of receiving the exposure they received. +One way to think about matching is as a crude "weight" where everyone who was matched gets a weight of 1 and everyone who was not matched gets a weight of 0 in the final sample. +Another option is to allow this weight to be smooth, applying a weight to allow, on average, the covariates of interest to be balanced in the weighted population. +To do this, we will construct a weight using the propensity score. +There are many different weights that can be applied, depending on your target estimand of interest (see @sec-estimands for details). +For this section, we will focus on the "Average Treatment Effect" weights, commonly referred to as an "inverse probability weight". +The weight is constructed as follows, where each observation is weighted by the *inverse* of the probability of receiving the exposure they received. $$w_{ATE} = \frac{X}{p} + \frac{(1 - X)}{1 - p}$$ -For example, if observation 1 had a very high likelihood of being exposed given their pre-exposure covariates ($p = 0.9$), but they in fact were *not* exposed, their weight would be 10 ($w_1 = 1 / (1 - 0.9)$). Likewise, if observation 2 had a very high likelihood of being exposed given their pre-exposure covariates ($p = 0.9$), and they *were* exposed, their weight would be 1.1 ($w_2 = 1 / 0.9$). Intuitively, we give *more* weight to observations who, based on their measured confounders, appear to have useful information for constructing a counterfactual -- we would have predicted that they were exposed and but by chance they were not, or vice-versa. The `{propensity}` package is useful for implementing propensity score weighting. +For example, if observation 1 had a very high likelihood of being exposed given their pre-exposure covariates ($p = 0.9$), but they in fact were *not* exposed, their weight would be 10 ($w_1 = 1 / (1 - 0.9)$). +Likewise, if observation 2 had a very high likelihood of being exposed given their pre-exposure covariates ($p = 0.9$), and they *were* exposed, their weight would be 1.1 ($w_2 = 1 / 0.9$). +Intuitively, we give *more* weight to observations who, based on their measured confounders, appear to have useful information for constructing a counterfactual -- we would have predicted that they were exposed and but by chance they were not, or vice-versa. +The `{propensity}` package is useful for implementing propensity score weighting. ```{r} library(propensity) @@ -82,7 +103,3 @@ seven_dwarfs_9_with_wt |> head() |> knitr::kable() ``` - - - - diff --git a/chapters/10-evaluating-ps.qmd b/chapters/10-evaluating-ps.qmd index 561b11b..fabef87 100644 --- a/chapters/10-evaluating-ps.qmd +++ b/chapters/10-evaluating-ps.qmd @@ -8,22 +8,27 @@ status("polishing") ``` -Propensity scores are inherently *balancing* scores. The goal is to *balance* the exposure groups across confounders. +Propensity scores are inherently *balancing* scores. +The goal is to *balance* the exposure groups across confounders. ## Calculating the standardized mean difference -One way to assess balance is the *standardized mean difference*. This measure helps you assess whether the average value for the confounder is balanced between exposure groups. For example, if you have some continuous confounder, $Z$, and $\bar{z}_{exposed} = \frac{\sum Z_i(X_i)}{\sum X_i}$ is the mean value of $Z$ among the exposed, $\bar{z}_{unexposed} = \frac{\sum Z_i(1-X_i)}{\sum 1-X_i}$ is the mean value of $Z$ among the unexposed, $s_{exposed}$ is the sample standard deviation of $Z$ among the exposed and $s_{unexposed}$ is the sample standard deviation of $Z$ among the unexposed, then the standardized mean difference can be expressed as follows: +One way to assess balance is the *standardized mean difference*. +This measure helps you assess whether the average value for the confounder is balanced between exposure groups. +For example, if you have some continuous confounder, $Z$, and $\bar{z}_{exposed} = \frac{\sum Z_i(X_i)}{\sum X_i}$ is the mean value of $Z$ among the exposed, $\bar{z}_{unexposed} = \frac{\sum Z_i(1-X_i)}{\sum 1-X_i}$ is the mean value of $Z$ among the unexposed, $s_{exposed}$ is the sample standard deviation of $Z$ among the exposed and $s_{unexposed}$ is the sample standard deviation of $Z$ among the unexposed, then the standardized mean difference can be expressed as follows: $$ d =\frac{\bar{z}_{exposed}-\bar{z}_{unexposued}}{\frac{\sqrt{s^2_{exposed}+s^2_{unexposed}}}{2}} -$$ -In the case of a binary $Z$ (a confounder with just two levels), $\bar{z}$ is replaced with the sample proportion in each group (e.g., $\hat{p}_{exposed}$ or $\hat{p}_{unexposed}$ ) and $s^2=\hat{p}(1-\hat{p})$. In the case where $Z$ is categorical with more than two categories, $\bar{z}$ is the vector of proportions of each category level within a group and the denominator is the multinomial covariance matrix ($S$ below), as the above can be written more generally as: +$$ In the case of a binary $Z$ (a confounder with just two levels), $\bar{z}$ is replaced with the sample proportion in each group (e.g., $\hat{p}_{exposed}$ or $\hat{p}_{unexposed}$ ) and $s^2=\hat{p}(1-\hat{p})$. +In the case where $Z$ is categorical with more than two categories, $\bar{z}$ is the vector of proportions of each category level within a group and the denominator is the multinomial covariance matrix ($S$ below), as the above can be written more generally as: $$ d = \sqrt{(\bar{z}_{exposed} - \bar{z}_{unexposed})^TS^{-1}(\bar{z}_{exposed} - \bar{z}_{unexposed})} $$ -Often, we calculate the standardized mean difference for each confounder in the full, unadjusted, data set and then compare this to an *adjusted* standardized mean difference. If the propensity score is incorporated using *matching*, this adjusted standardized mean difference uses the exact equation as above, but restricts the sample considered to only those that were matched. If the propensity score is incorporated using *weighting*, this adjusted standardized mean difference *weights* each of the above components using the constructed propensity score weight. +Often, we calculate the standardized mean difference for each confounder in the full, unadjusted, data set and then compare this to an *adjusted* standardized mean difference. +If the propensity score is incorporated using *matching*, this adjusted standardized mean difference uses the exact equation as above, but restricts the sample considered to only those that were matched. +If the propensity score is incorporated using *weighting*, this adjusted standardized mean difference *weights* each of the above components using the constructed propensity score weight. In R, the `{halfmoon}` package has a function `tidy_smd` that will calculate this for a data set. @@ -39,7 +44,7 @@ smds <- tidy_smd( ) ``` -Let's look at an example using the same data as @sec-using-ps. +Let's look at an example using the same data as @sec-using-ps. ```{r} library(broom) @@ -76,13 +81,16 @@ smds For example, we see above that the *observed* standardized mean difference (prior to incorporating the propensity score) for ticket season is `r smds |> filter(variable == "park_ticket_season" & method == "observed") |> pull(smd) |> round(2)`, however after incorporating the propensity score weight this is attenuated, now `r smds |> filter(variable == "park_ticket_season" & method == "w_ate") |> pull(smd) |> round(2)`. -One downside of this metric is it only quantifying balance *on the mean*, which may not be sufficient for continuous confounders, as it is possible to be balanced on the mean but severely imbalanced in the tails. At the end of this chapter we will show you a few tools for examining balance across the full distribution of the confounder. +One downside of this metric is it only quantifying balance *on the mean*, which may not be sufficient for continuous confounders, as it is possible to be balanced on the mean but severely imbalanced in the tails. +At the end of this chapter we will show you a few tools for examining balance across the full distribution of the confounder. ## Visualizing balance ### Love Plots -Let's start by visualizing these standardized mean differences. To do so, we like to use a *Love Plot* (named for Thomas Love, as he was one of the first to popularize them). The `{halfmoon}` package has a function `geom_love` that simplifies this implementation. +Let's start by visualizing these standardized mean differences. +To do so, we like to use a *Love Plot* (named for Thomas Love, as he was one of the first to popularize them). +The `{halfmoon}` package has a function `geom_love` that simplifies this implementation. ```{r} ggplot( @@ -97,10 +105,14 @@ ggplot( geom_love() ``` - ### Boxplots and eCDF plots -As mentioned above, one issue with the standardized mean differences is they only quantify balance on a single point for continuous confounders (the mean). It can be helpful to visualize the whole distribution to ensure that there is not residual imbalance in the tails. Let's first use a boxplot. As an example, let's use the `park_temperature_high` variable. When we make boxplots, we prefer to always jitter the points on top to make sure we aren't masking and data anomolies -- we use `geom_jitter` to accomplish this. First, we will make the unweighted boxplot. +As mentioned above, one issue with the standardized mean differences is they only quantify balance on a single point for continuous confounders (the mean). +It can be helpful to visualize the whole distribution to ensure that there is not residual imbalance in the tails. +Let's first use a boxplot. +As an example, let's use the `park_temperature_high` variable. +When we make boxplots, we prefer to always jitter the points on top to make sure we aren't masking and data anomolies -- we use `geom_jitter` to accomplish this. +First, we will make the unweighted boxplot. ```{r} #| label: fig-boxplot @@ -142,7 +154,8 @@ ggplot( ) ``` -Similarly, we can also examine the empirical cumulative distribution function (eCDF) for the confounder stratified by each exposure group. The unweighted eCDF can be visualized using `geom_ecdf` +Similarly, we can also examine the empirical cumulative distribution function (eCDF) for the confounder stratified by each exposure group. +The unweighted eCDF can be visualized using `geom_ecdf` ```{r} #| label: fig-ecdf @@ -192,7 +205,15 @@ ggplot( ) ``` -Examining @fig-weighted-ecdf, we can notice a few things. First, compared to @fig-ecdf there is improvement in the overlap between the two distributions. In @fig-ecdf, the green line is almost always noticeably above the purple, whereas in @fig-weighted-ecdf the two lines appear to mostly overlap until we reach slightly above 80 degrees. After 80 degrees, the lines appear to diverge in the weighted plot. This is why it can be useful to examine the full distribution rather than a single summary measure. If we had just used the standardized mean difference, for example, we would have likely said these two groups are balanced and moved on. Looking at @fig-weighted-ecdf suggests that perhaps there is a non-linear relationship between the probability of having an extra magic morning and the historic high temperature. Let's try refitting our propensity score model using a natural spline. We can use the function `splines::ns` for this. +Examining @fig-weighted-ecdf, we can notice a few things. +First, compared to @fig-ecdf there is improvement in the overlap between the two distributions. +In @fig-ecdf, the green line is almost always noticeably above the purple, whereas in @fig-weighted-ecdf the two lines appear to mostly overlap until we reach slightly above 80 degrees. +After 80 degrees, the lines appear to diverge in the weighted plot. +This is why it can be useful to examine the full distribution rather than a single summary measure. +If we had just used the standardized mean difference, for example, we would have likely said these two groups are balanced and moved on. +Looking at @fig-weighted-ecdf suggests that perhaps there is a non-linear relationship between the probability of having an extra magic morning and the historic high temperature. +Let's try refitting our propensity score model using a natural spline. +We can use the function `splines::ns` for this. diff --git a/chapters/11-estimands.qmd b/chapters/11-estimands.qmd index 9f07a22..174b6e2 100644 --- a/chapters/11-estimands.qmd +++ b/chapters/11-estimands.qmd @@ -1065,7 +1065,8 @@ Instead of worrying about which version of the odds ratio is right, we recommend Much ink has been spilled about the odds ratio versus the risk ratio and the relative versus absolute scale. We suggest that you present all three measures (the odds ratio, the risk ratio, and the risk difference) together with the baseline probability of the outcome. Each offers a different perspective on the causal effect. -Be careful to interpret them with regard to the treatment group that you've included in your estimate. For example, the average risk difference calculated with ATT weights is the average risk difference among the treated. +Be careful to interpret them with regard to the treatment group that you've included in your estimate. +For example, the average risk difference calculated with ATT weights is the average risk difference among the treated. ::: callout-note ## The linear probability model diff --git a/chapters/12-outcome-model.qmd b/chapters/12-outcome-model.qmd index 406036e..a372a6a 100644 --- a/chapters/12-outcome-model.qmd +++ b/chapters/12-outcome-model.qmd @@ -10,7 +10,8 @@ status("polishing") ## Using matched data sets -When fitting an outcome model on matched data sets, we can simply subset the original data to only those who were matched and then fit a model on these data as we would otherwise. For example, re-performing the matching as we did in @sec-using-ps, we can extract the matched observations in a dataset called `matched_data` as follows. +When fitting an outcome model on matched data sets, we can simply subset the original data to only those who were matched and then fit a model on these data as we would otherwise. +For example, re-performing the matching as we did in @sec-using-ps, we can extract the matched observations in a dataset called `matched_data` as follows. ```{r} #| message: false @@ -29,18 +30,22 @@ m <- matchit( matched_data <- get_matches(m) ``` -We can then fit the outcome model on this data. For this analysis, we are interested in the impact of extra magic morning hours on the average posted wait time between 9 and 10am. The linear model below will estimate this in the matched cohort. +We can then fit the outcome model on this data. +For this analysis, we are interested in the impact of extra magic morning hours on the average posted wait time between 9 and 10am. +The linear model below will estimate this in the matched cohort. ```{r} lm(wait_minutes_posted_avg ~ park_extra_magic_morning, data = matched_data) |> tidy(conf.int = TRUE) ``` -Recall that by default `{MatchIt}` estimates an average treatment effect among the treated. This means among days that have extra magic hours, the expected impact of having extra magic hours on the average posted wait time between 9 and 10am is 7.9 minutes (95% CI: 1.2-14.5). +Recall that by default `{MatchIt}` estimates an average treatment effect among the treated. +This means among days that have extra magic hours, the expected impact of having extra magic hours on the average posted wait time between 9 and 10am is 7.9 minutes (95% CI: 1.2-14.5). ## Using weights in outcome models -Now let's use propensity score weights to estimate this same estimand. We will use the ATT weights so the analysis matches that for matching above. +Now let's use propensity score weights to estimate this same estimand. +We will use the ATT weights so the analysis matches that for matching above. ```{r} #| message: false @@ -58,7 +63,7 @@ seven_dwarfs_9_with_wt <- seven_dwarfs_9_with_ps |> mutate(w_att = wt_att(.fitted, park_extra_magic_morning)) ``` -We can fit a *weighted* outcome model by using the `weights` argument. +We can fit a *weighted* outcome model by using the `weights` argument. ```{r} lm( @@ -69,10 +74,10 @@ lm( tidy() ``` -Using weighting, we estimate that among days that have extra magic hours, the expected impact of having extra magic hours on the average posted wait time between 9 and 10am is 6.2 minutes. +Using weighting, we estimate that among days that have extra magic hours, the expected impact of having extra magic hours on the average posted wait time between 9 and 10am is 6.2 minutes. While this approach will get us the desired estimate for the point estimate, the default output using the `lm` function for the uncertainty (the standard errors and confidence intervals) are not correct. -::: {.callout-tip} +::: callout-tip ## Causal inference with `group_by()` and `summarize()`, revisted For this simple example, the weighted outcome model is equivalent to taking the difference in the weighted means. @@ -87,24 +92,26 @@ wt_means The difference is `r round(wt_means$average_wait[[2]] - wt_means$average_wait[[1]], 2)`, the same as the weighted outcome model. -The weighted population is a psuedo-population where there is no confounding by the variables in the propensity score. Philosophically and practically, we can make calculations with the data from this population. Causal inference with `group_by()` and `summarize()` works just fine now, since we've already accounted for confounding in the weights. +The weighted population is a psuedo-population where there is no confounding by the variables in the propensity score. +Philosophically and practically, we can make calculations with the data from this population. +Causal inference with `group_by()` and `summarize()` works just fine now, since we've already accounted for confounding in the weights. ::: - ## Estimating uncertainty There are three ways to estimate the uncertainty: -1. A bootstrap -2. A sandwich estimator that only takes into account the outcome model -3. A sandwich estimator that takes into account the propensity score model +1. A bootstrap +2. A sandwich estimator that only takes into account the outcome model +3. A sandwich estimator that takes into account the propensity score model The first option can be computationally intensive, but should get you the correct estimates. -The second option is computationally the easiest, but tends to overestimate the variability. There are not many current solutions in R (aside from coding it up yourself) for the third; however, the `{PSW}` package will do this. +The second option is computationally the easiest, but tends to overestimate the variability. +There are not many current solutions in R (aside from coding it up yourself) for the third; however, the `{PSW}` package will do this. ### The bootstrap -1. Create a function to run your analysis once on a sample of your data +1. Create a function to run your analysis once on a sample of your data ```{r} fit_ipw <- function(.split, ...) { @@ -137,8 +144,7 @@ fit_ipw <- function(.split, ...) { } ``` - -2. Use {rsample} to bootstrap our causal effect +2. Use {rsample} to bootstrap our causal effect ```{r} #| message: false @@ -158,7 +164,6 @@ ipw_results <- bootstrapped_seven_dwarfs |> ipw_results ``` - Let's look at the results. ```{r} @@ -176,8 +181,7 @@ ipw_results |> theme_minimal() ``` - -3. Pull out the causal effect +3. Pull out the causal effect ```{r} # get t-based CIs @@ -186,11 +190,13 @@ boot_estimate <- int_t(ipw_results, boot_fits) |> boot_estimate ``` -We estimate that among days that have extra magic hours, the expected impact of having extra magic hours on the average posted wait time between 9 and 10am is `r round(boot_estimate$.estimate, 1)` minutes, 95% CI (`r round(boot_estimate$.lower, 1)`, `r round(boot_estimate$.upper, 1)`). +We estimate that among days that have extra magic hours, the expected impact of having extra magic hours on the average posted wait time between 9 and 10am is `r round(boot_estimate$.estimate, 1)` minutes, 95% CI (`r round(boot_estimate$.lower, 1)`, `r round(boot_estimate$.upper, 1)`). ### The outcome model sandwich -There are two ways to get the sandwich estimator. The first is to use the same weighted outcome model as above along with the `{sandwich}` package. Using the `sandwich` function, we can get the robust estimate for the variance for the parameter of interest, as shown below. +There are two ways to get the sandwich estimator. +The first is to use the same weighted outcome model as above along with the `{sandwich}` package. +Using the `sandwich` function, we can get the robust estimate for the variance for the parameter of interest, as shown below. ```{r} #| message: false @@ -205,7 +211,8 @@ weighted_mod <- lm( sandwich(weighted_mod) ``` -Here, our robust variance estimate is `r round(sandwich(weighted_mod)[2,2], 3)`. We can then use this to construct a robust confidence interval. +Here, our robust variance estimate is `r round(sandwich(weighted_mod)[2,2], 3)`. +We can then use this to construct a robust confidence interval. ```{r} robust_var <- sandwich(weighted_mod)[2, 2] @@ -215,9 +222,11 @@ ub <- point_est + 1.96 * sqrt(robust_var) lb ub ``` -We estimate that among days that have extra magic hours, the expected impact of having extra magic hours on the average posted wait time between 9 and 10am is `r round(point_est, 1)` minutes, 95% CI (`r round(lb, 1)`, `r round(ub, 1)`). -Alternatively, we could fit the model using the `{survey}` package. To do this, we need to create a design object, like we did when fitting weighted tables. +We estimate that among days that have extra magic hours, the expected impact of having extra magic hours on the average posted wait time between 9 and 10am is `r round(point_est, 1)` minutes, 95% CI (`r round(lb, 1)`, `r round(ub, 1)`). + +Alternatively, we could fit the model using the `{survey}` package. +To do this, we need to create a design object, like we did when fitting weighted tables. ```{r} #| message: false @@ -237,9 +246,13 @@ Then we can use `svyglm` to fit the outcome model. svyglm(wait_minutes_posted_avg ~ park_extra_magic_morning, des) |> tidy(conf.int = TRUE) ``` + ### Sandwich estimator that takes into account the propensity score model -The correct sandwich estimator will also take into account the propensity score model. The `{PSW}` will allow us to do this. This package has some quirks, for example it doesn't work well with categorical variables, so we need to create dummy variables for `park_ticket_season` to pass into the model. *Actually, the code below isn't working because it seems there is a bug in the package. Stay tuned!* +The correct sandwich estimator will also take into account the propensity score model. +The `{PSW}` will allow us to do this. +This package has some quirks, for example it doesn't work well with categorical variables, so we need to create dummy variables for `park_ticket_season` to pass into the model. +*Actually, the code below isn't working because it seems there is a bug in the package. Stay tuned!* ```{r} #| eval: false @@ -258,4 +271,3 @@ psw( out.var = "wait_minutes_posted_avg" ) ``` - diff --git a/chapters/15-g-comp.qmd b/chapters/15-g-comp.qmd index 82e3e5b..efb4249 100644 --- a/chapters/15-g-comp.qmd +++ b/chapters/15-g-comp.qmd @@ -8,27 +8,27 @@ status("wip") ``` - ## The Parametric G-Formula Let's pause to recap a typical goal of the causal analyses we've seen in this book so far: to estimate what would happen if *everyone* in the study were exposed versus what would happen if *no one* was exposed. To do this, we've used weighting techniques that create confounder-balanced pseudopopulations which, in turn, give rise to unbiased causal effect estimates in marginal outcome models. One alternative approach to weighting is called the parametric G-formula, which is generally executed through the following 4 steps: -1. Draw the appropriate time ordered DAG (as described in @sec-dags). +1. Draw the appropriate time ordered DAG (as described in @sec-dags). -2. For each point in time after baseline, decide on a parametric model that predicts each variable's value based on previously measured variables in the DAG. -These are often linear models for continuous variables or logistic regressions for binary variables. +2. For each point in time after baseline, decide on a parametric model that predicts each variable's value based on previously measured variables in the DAG. + These are often linear models for continuous variables or logistic regressions for binary variables. -3. Starting with a sample from the observed distribution of data at baseline, generate values for all subsequent variables according to the models in step 2 (i.e. conduct a *Monte Carlo simulation*). Do this with one key modification: for each exposure regime you are interested in comparing (e.g. everyone exposed versus everyone unexposed), assign the exposure variables accordingly (that is, don't let the simulation assign values for exposure variables). +3. Starting with a sample from the observed distribution of data at baseline, generate values for all subsequent variables according to the models in step 2 (i.e. conduct a *Monte Carlo simulation*). + Do this with one key modification: for each exposure regime you are interested in comparing (e.g. everyone exposed versus everyone unexposed), assign the exposure variables accordingly (that is, don't let the simulation assign values for exposure variables). -4. Compute the causal contrast of interest based on the simulated outcome in each exposure group. +4. Compute the causal contrast of interest based on the simulated outcome in each exposure group. ::: callout-tip ## Monte Carlo simulations Monte Carlo simulations are computational approaches that generate a sample of outcomes for random processes. -One example would be to calculate the probability of rolling "snake eyes" (two ones) on a single roll of two six-sided dice. +One example would be to calculate the probability of rolling "snake eyes" (two ones) on a single roll of two six-sided dice. We could certainly calculate this probability mathematically ($\frac{1}{6}*\frac{1}{6}=\frac{1}{36}\approx 2.8$%), though it can be just as quick to write a Monte Carlo simulation of the process (1,000,000 rolls shown below). ```{r} @@ -41,7 +41,8 @@ tibble( pull() |> sum() / n ``` -Monte Carlo simulations are extremely useful for estimating outcomes of complex processes for which closed mathematical solutions are not easy to determine. + +Monte Carlo simulations are extremely useful for estimating outcomes of complex processes for which closed mathematical solutions are not easy to determine. Indeed, that's why Monte Carlo simulations are so useful for the real-world causal mechanisms described in this book! ::: @@ -51,7 +52,8 @@ Recall in @sec-outcome-model that we estimated the impact of extra magic morning To do so, we fit a propensity score model for the exposure (`park_extra_magic_morning`) with the confounders `park_ticket_season`, `park_close`, and `park_temperature_high`. In turn, these propensity scores were converted to regression weights for the outcome model, which concluded that the expected impact of having extra magic hours on the average posted wait time between 9 and 10am is 6.2 minutes. -We will now reproduce this analysis, instead adopting the g-formula approach. Proceeding through the 4 steps outlined above, we begin by revisiting the time ordered DAG relevant to this question. +We will now reproduce this analysis, instead adopting the g-formula approach. +Proceeding through the 4 steps outlined above, we begin by revisiting the time ordered DAG relevant to this question. ```{r} #| label: fig-dag-magic-hours-wait-take-2 @@ -115,8 +117,8 @@ dagify( ) ``` -The second step is to specify a parametric model for each non-baseline variable that is based upon previously measured variables in the DAG. -This particular example is simple, since we only have two variables that are affected by previous features (`park_extra_magic_morning` and `wait_minutes_posted_avg`). +The second step is to specify a parametric model for each non-baseline variable that is based upon previously measured variables in the DAG. +This particular example is simple, since we only have two variables that are affected by previous features (`park_extra_magic_morning` and `wait_minutes_posted_avg`). Let's suppose that adequate models for these two variables are the simple logistic and linear models that follow. Of note, we're not yet going to use the model for the exposure (`park_extra_magic_morning`), but we're including the step here because it will be an important part of patterns you will see in the next section (@sec-dynamic). @@ -158,9 +160,9 @@ df_sim_baseline <- seven_dwarfs_9 |> sample_n(10000, replace = TRUE) ``` -With this population in hand, we can now simulate what would happen at each subsequent time point according to the parametric models we just defined. -Remember that, in step 3, an important caveat is that for the variable upon which we wish to intervene (in this case, `park_extra_magic_morning`) we don't need to let the model determine the values; rather, we set them. -Specifically, we'll set the first 5000 to `park_extra_magic_morning = 1` and the second 5000 to `park_extra_magic_morning = 0`. +With this population in hand, we can now simulate what would happen at each subsequent time point according to the parametric models we just defined. +Remember that, in step 3, an important caveat is that for the variable upon which we wish to intervene (in this case, `park_extra_magic_morning`) we don't need to let the model determine the values; rather, we set them. +Specifically, we'll set the first 5000 to `park_extra_magic_morning = 1` and the second 5000 to `park_extra_magic_morning = 0`. Other simulations (in this case, the only remaining variable, `wait_minutes_posted_avg`) proceed as expected. ```{r} @@ -175,7 +177,7 @@ df_outcome <- fit_wait_minutes |> ``` All that is left to do is compute the causal contrast we wish to estimate. -Here, that contrast is the difference between expected wait minutes on extra magic mornings versus mornings without the extra magic program. +Here, that contrast is the difference between expected wait minutes on extra magic mornings versus mornings without the extra magic program. ```{r} df_outcome |> @@ -187,12 +189,11 @@ We see that the difference, $74.3-68.1=6.2$ is the same as our estimate of 6.2 w ## The g-formula for continuous exposures -As previously mentioned, a key strength of the g-formula is its capacity to handle continuous exposures, a situation in which IP weighting can give rise to unstable estimates. +As previously mentioned, a key strength of the g-formula is its capacity to handle continuous exposures, a situation in which IP weighting can give rise to unstable estimates. Here, we briefly repeat the example from @sec-continuous-exposures to show how this is done. To extend the pattern, we will wrap this execution of the technique in a bootstrap to show how confidence intervals are computed. -Recall, our causal question of interest is "Do posted wait times for the Seven Dwarfs Mine Train at 8 am affect actual wait times at 9 am?" -The time-ordered DAG for this question (step 1) is: +Recall, our causal question of interest is "Do posted wait times for the Seven Dwarfs Mine Train at 8 am affect actual wait times at 9 am?" The time-ordered DAG for this question (step 1) is: ```{r} #| label: fig-dag-avg-wait-2 @@ -305,7 +306,7 @@ fit_models <- function(.data) { } ``` -Next, we write a function which will complete step 3: from a random sample from the distribution of baseline variables, generate values for all subsequent variables (except the intervention variable) according to the models we defined. +Next, we write a function which will complete step 3: from a random sample from the distribution of baseline variables, generate values for all subsequent variables (except the intervention variable) according to the models we defined. ```{r} # The arguments to simulate_process are as follows: @@ -355,7 +356,7 @@ simulate_process <- function( } ``` -Finally, in step 4, we compute the summary statistics and causal contrast of interest using our simulated data. +Finally, in step 4, we compute the summary statistics and causal contrast of interest using our simulated data. ```{r} # sim_obj is a list created by our simulate_process() function @@ -381,7 +382,7 @@ compute_stats <- function(sim_obj) { } ``` -Now, let's put it all together to get a single point estimate. +Now, let's put it all together to get a single point estimate. Once we've seen that in action, we'll bootstrap for a confidence interval. ```{r} @@ -433,7 +434,8 @@ results <- int_pctl(boots, models) results ``` -In summary, our results are intepreted as follows: setting the posted wait time at 8am to 60 minutes results in an actual wait time at 9am of `r results |> filter(term == "x_60") |> pull(.estimate) |> round()`, while setting the posted wait time to 30 minutes gives a longer wait time of `r results |> filter(term == "x_30") |> pull(.estimate) |> round()`. Put another way, increasing the posted wait time at 8am from 30 minutes to 60 minutes results in a `r round(-results[3,3])` minute shorter wait time at 9am. +In summary, our results are intepreted as follows: setting the posted wait time at 8am to 60 minutes results in an actual wait time at 9am of `r results |> filter(term == "x_60") |> pull(.estimate) |> round()`, while setting the posted wait time to 30 minutes gives a longer wait time of `r results |> filter(term == "x_30") |> pull(.estimate) |> round()`. +Put another way, increasing the posted wait time at 8am from 30 minutes to 60 minutes results in a `r round(-results[3,3])` minute shorter wait time at 9am. Note that one of our models threw a warning regarding perfect discrimination (`fitted probabilities numerically 0 or 1 occurred`); this can happen when we don't have a large sample size and one of our models is overspecified due to complexity. In this exercise, the flexibility added by the spline in the regression for `wait_minutes_actual_avg` is what caused the issue. @@ -442,5 +444,4 @@ We've left the warning here to highlight a common challenge that needs resolutio ## Dynamic treatment regimes with the g-formula {#sec-dynamic} - ## The Natural Course diff --git a/chapters/16-interaction.qmd b/chapters/16-interaction.qmd index 18300c7..90412c3 100644 --- a/chapters/16-interaction.qmd +++ b/chapters/16-interaction.qmd @@ -13,4 +13,5 @@ status("unstarted") ## Fitting interaction terms in causal models + diff --git a/chapters/17-missingness-and-measurement.qmd b/chapters/17-missingness-and-measurement.qmd index 1be0b60..d7cd442 100644 --- a/chapters/17-missingness-and-measurement.qmd +++ b/chapters/17-missingness-and-measurement.qmd @@ -36,7 +36,8 @@ In the TouringPlans data, most variables are complete and likely accurately meas One variable with missingness and measurement error is the actual wait times for rides. If you recall, the data relies on humans to wait in line, either consumers of the data who report their experience or people hired to wait in line to report the wait time. Thus, missingness is primarily related to whether someone is there to measure the wait time. -When someone is there to measure it, it's also likely measured with error, and that error likely depends on who the person is. For example, a visitor to Disney World estimating their wait time and submitting it to TouringPlans is likely producing a value with more error than someone paid to stay in line and count the minutes. +When someone is there to measure it, it's also likely measured with error, and that error likely depends on who the person is. +For example, a visitor to Disney World estimating their wait time and submitting it to TouringPlans is likely producing a value with more error than someone paid to stay in line and count the minutes. That said, let's take measurement error and missingness one at a time. @@ -229,7 +230,8 @@ This is sometimes called *independent, non-differential* measurement error: the ::: callout-warning As the correlation approaches 0, measurement becomes random for the relationship under study. This often means that when variables are measured with independent measurement error, the relationship of the measured values approaches null even if there is an arrow between the true values. -Here, the coefficient of `x` should be about 1, but as the random measurement worsens, the coefficient gets closer to 0. There's no relationship between the randomness (`u`) induced by mismeasurement and `y`. +Here, the coefficient of `x` should be about 1, but as the random measurement worsens, the coefficient gets closer to 0. +There's no relationship between the randomness (`u`) induced by mismeasurement and `y`. ```{r} n <- 1000 diff --git a/chapters/18-longitudinal.qmd b/chapters/18-longitudinal.qmd index 0269669..0f88c7a 100644 --- a/chapters/18-longitudinal.qmd +++ b/chapters/18-longitudinal.qmd @@ -15,4 +15,3 @@ rnorm(5) ``` ## Time-varying confounding and exposures - diff --git a/chapters/21-sensitivity.qmd b/chapters/21-sensitivity.qmd index 7bc4752..6b4e732 100644 --- a/chapters/21-sensitivity.qmd +++ b/chapters/21-sensitivity.qmd @@ -203,13 +203,14 @@ The reverse is also true: the causal structure of your research question also im One way that researchers take advantage of this implication is through *negative controls*. A negative control is either an exposure (negative exposure control) or outcome (negative outcome control) similar to your question in as many ways as possible, except that there *shouldn't* be a causal effect. @Lipsitch2010 describe negative controls for observational research. -In their article, they reference standard controls in bench science. In a lab experiment, any of these actions should lead to a null effect: +In their article, they reference standard controls in bench science. +In a lab experiment, any of these actions should lead to a null effect: 1. Leave out an essential ingredient. 2. Inactivate the hypothesized active ingredient. 3. Check for an effect that would be impossible by the hypothesized outcome. -There's nothing unique to lab work here; these scientists merely probe the logical implications of their understanding and hypotheses. +There's nothing unique to lab work here; these scientists merely probe the logical implications of their understanding and hypotheses. To find a good negative control, you usually need to extend your DAG to include more of the causal structure surrounding your question. Let's look at some examples. @@ -777,15 +778,19 @@ We can take this further using quantitative bias analysis, which uses mathematic ### Sensitivity analyses for unmeasured confounders -Sensitivity analyses for unmeasured confounding are important tools in observational studies to assess how robust findings are to potential unmeasured factors [@d2022sensitivity]. These analyses rely on three key components: +Sensitivity analyses for unmeasured confounding are important tools in observational studies to assess how robust findings are to potential unmeasured factors [@d2022sensitivity]. +These analyses rely on three key components: -1) the observed exposure-outcome effect after adjusting for measured confounders, -2) the estimated relationship between a hypothetical unmeasured confounder and the exposure, and -3) the estimated relationship between that unmeasured confounder and the outcome. +1) the observed exposure-outcome effect after adjusting for measured confounders,\ +2) the estimated relationship between a hypothetical unmeasured confounder and the exposure, and\ +3) the estimated relationship between that unmeasured confounder and the outcome. -By specifying plausible values for these relationships, researchers can quantify how much the observed effect might change if such an unmeasured confounder existed. +By specifying plausible values for these relationships, researchers can quantify how much the observed effect might change if such an unmeasured confounder existed. -Let's think about why this works in the context of our example above. Let's assume @fig-dag-magic-sens displays the true relationship between our exposure and outcome of interest. Suppose we hadn't measured historic high temperature, denoted by the dashed lines; we have an unmeasured confounder. The three key components above are described by 1) the arrow between 'Extra Magic Morning' and 'Average wait', 2) The arrow between 'Historic high temperature' and 'Extra Magic Morning', and 3) the arrow between 'Historic high temperature' and 'Average wait'. +Let's think about why this works in the context of our example above. +Let's assume @fig-dag-magic-sens displays the true relationship between our exposure and outcome of interest. +Suppose we hadn't measured historic high temperature, denoted by the dashed lines; we have an unmeasured confounder. +The three key components above are described by 1) the arrow between 'Extra Magic Morning' and 'Average wait', 2) The arrow between 'Historic high temperature' and 'Extra Magic Morning', and 3) the arrow between 'Historic high temperature' and 'Average wait'. ```{r} #| label: fig-dag-magic-sens @@ -821,125 +826,103 @@ emm_wait_dag |> ) ``` -Various methods are available depending on the type of outcome (e.g. continuous, binary, time-to-event) and how much is known about potential unmeasured confounders. While these analyses cannot prove the absence of unmeasured confounding, they provide valuable insight into how sensitive results are to violations of the "no unmeasured confounders" assumption that is crucial for causal inference in observational studies. +Various methods are available depending on the type of outcome (e.g. continuous, binary, time-to-event) and how much is known about potential unmeasured confounders. +While these analyses cannot prove the absence of unmeasured confounding, they provide valuable insight into how sensitive results are to violations of the "no unmeasured confounders" assumption that is crucial for causal inference in observational studies. #### Observed exposure-outcome effect -The first componenet, the observed exposure-outcome effect, is the proposed causal effect of interest, i.e. the effect you would like to perform a sensitivity analysis on. The effect itself will depend on the choice of outcome model, which in turn often depends on the distribution of the outcome and the desired effect measure: +The first componenet, the observed exposure-outcome effect, is the proposed causal effect of interest, i.e. the effect you would like to perform a sensitivity analysis on. +The effect itself will depend on the choice of outcome model, which in turn often depends on the distribution of the outcome and the desired effect measure: -1. For continuous outcomes: Linear models or generalized linear models (GLMs) with Gaussian distribution and identity link are used, typically estimating a coefficient. +1. For continuous outcomes: Linear models or generalized linear models (GLMs) with Gaussian distribution and identity link are used, typically estimating a coefficient. -2. For binary outcomes, we have a few choices: +2. For binary outcomes, we have a few choices: -* GLMs with binomial distribution and log link -* GLMs with Poisson distribution and log link -* GLMs with binomial distribution and logit link -These estimate coefficients, which can be exponentiated to obtain risk ratios (log link models) or odds ratios (logit link models). +- GLMs with binomial distribution and log link +- GLMs with Poisson distribution and log link +- GLMs with binomial distribution and logit link These estimate coefficients, which can be exponentiated to obtain risk ratios (log link models) or odds ratios (logit link models). +3. For time-to-event outcomes: Cox proportional hazards models are used, with the hazard ratio obtained by exponentiating the coefficient. -3. For time-to-event outcomes: Cox proportional hazards models are used, with the hazard ratio obtained by exponentiating the coefficient. - -Let's use the analysis from @tbl-alt-sets where we only adjusted for 'Time park closed' and 'Ticket season'. According to @fig-dag-magic-sens, we know 'Historic high temperature' is also a confounder, but it is *unmeasured* so we cannot include it in our practical adjustment set. This resulted in an observed effect of `r round(effects[2], 2)`. +Let's use the analysis from @tbl-alt-sets where we only adjusted for 'Time park closed' and 'Ticket season'. +According to @fig-dag-magic-sens, we know 'Historic high temperature' is also a confounder, but it is *unmeasured* so we cannot include it in our practical adjustment set. +This resulted in an observed effect of `r round(effects[2], 2)`. #### Unmeasured confounder-exposure effect The relationship between an unmeasured confounder and the exposure can be characterized in three ways: -1. For a binary unmeasured confounder: +1. For a binary unmeasured confounder: -* Prevalence of the unmeasured confounder in the exposed group -* Prevalence of the unmeasured confounder in the unexposed group +- Prevalence of the unmeasured confounder in the exposed group +- Prevalence of the unmeasured confounder in the unexposed group -2. For a continuous unmeasured confounder (assuming a normal distribution and unit variance): +2. For a continuous unmeasured confounder (assuming a normal distribution and unit variance): -* Difference in means of the unmeasured confounder between exposed and unexposed groups +- Difference in means of the unmeasured confounder between exposed and unexposed groups -3. Distribution-agnostic approach: +3. Distribution-agnostic approach: -* Partial $R^2$, representing the proportion of variation in the exposure explained by the unmeasured confounder after accounting for measured confounders +- Partial $R^2$, representing the proportion of variation in the exposure explained by the unmeasured confounder after accounting for measured confounders These characterizations allow researchers to specify the unmeasured confounder-exposure relationship in sensitivity analyses, accommodating different types of confounders and levels of knowledge about their distribution. -Our unmeasured confounder here, 'Historic high temperature', is continuous. For this example, let's assume it is normally distributed. We say to assume "unit variance" (a variance of 1), because it makes it easier to talk about the impact of the confounder in standard-deviation terms. Let's assume that on days with extra magic morning hours the historic high temperature is normally distributed with a mean of 80.5 degrees and a standard deviation of 9 degrees. Likewise, assume that on days without extra magic morning hours the historic high temperature is normally distributed with a mean of 82 degrees and a standard deviation of 9 degrees. We can convert these to 'unit variance' normally distributed variables by dividing by that standard deviation, 9 (sometimes we refer to this as *standardizing* our variable); this gives us a standardized mean of 8.94 for days with extra magic morning hours and 9.11 for the others, or a mean difference of -0.17. Hold on to this number; we'll use it in conjunction with the next section for our sensitivity analysis. +Our unmeasured confounder here, 'Historic high temperature', is continuous. +For this example, let's assume it is normally distributed. +We say to assume "unit variance" (a variance of 1), because it makes it easier to talk about the impact of the confounder in standard-deviation terms. +Let's assume that on days with extra magic morning hours the historic high temperature is normally distributed with a mean of 80.5 degrees and a standard deviation of 9 degrees. +Likewise, assume that on days without extra magic morning hours the historic high temperature is normally distributed with a mean of 82 degrees and a standard deviation of 9 degrees. +We can convert these to 'unit variance' normally distributed variables by dividing by that standard deviation, 9 (sometimes we refer to this as *standardizing* our variable); this gives us a standardized mean of 8.94 for days with extra magic morning hours and 9.11 for the others, or a mean difference of -0.17. +Hold on to this number; we'll use it in conjunction with the next section for our sensitivity analysis. #### Unmeasured confounder-outcome effect The relationship between an unmeasured confounder and the outcome can be quantified in two main ways: -1. Coefficient-based approach: Estimate the coefficient for an unmeasured confounder in a fully adjusted outcome model. You can also estimate the exponentiated coefficient (risk ratio, odds ratio, or hazard ratio) - -2. Distribution-agnostic approach (for continuous outcomes): Use partial $R^2$, representing the proportion of variation in the outcome explained by the unmeasured confounder after accounting for the exposure and measured confounders +1. Coefficient-based approach: Estimate the coefficient for an unmeasured confounder in a fully adjusted outcome model. + You can also estimate the exponentiated coefficient (risk ratio, odds ratio, or hazard ratio) -Let's do the coeffient-based approach. In our case, we need to estimate what we think the coefficient bewteen our standardized 'Historic high temperature' variable and our outcome after adjusting for our exposure as well as the other measured confounders (in this case ticket season and the time the park closed). Another way to describe this effect in the context of this problem is: "How would the average posted wait time change if we changed the historic high temperature by one standard deviation, after adjusting for whether there were extra magic morning hours, the park close time, and the ticket season?" Let's suppose we think this would change by -2.3 minutes. That is, if the historic high temperature is one standard deviation unit higher (in our scenario, 9 degrees warmer), we expect this to decrease the average posted wait time by 2.3 minutes. +2. Distribution-agnostic approach (for continuous outcomes): Use partial $R^2$, representing the proportion of variation in the outcome explained by the unmeasured confounder after accounting for the exposure and measured confounders +Let's do the coeffient-based approach. +In our case, we need to estimate what we think the coefficient bewteen our standardized 'Historic high temperature' variable and our outcome after adjusting for our exposure as well as the other measured confounders (in this case ticket season and the time the park closed). +Another way to describe this effect in the context of this problem is: "How would the average posted wait time change if we changed the historic high temperature by one standard deviation, after adjusting for whether there were extra magic morning hours, the park close time, and the ticket season?" Let's suppose we think this would change by -2.3 minutes. +That is, if the historic high temperature is one standard deviation unit higher (in our scenario, 9 degrees warmer), we expect this to decrease the average posted wait time by 2.3 minutes. For a mathematical explanation of these quantities, see @d2022sensitivity. #### Putting the components together -Once you have estimated the above three quantities, we can calculate a updated effect estimate between the exposure and outcome that takes into account an unmeasured factor like the one specified. We can use the {tipr} R package to perform these analyses. The functions in the {tipr} package follow a unified grammar. The function names follow this form: `{action}_{effect}_with_{what}`. - -For example, to adjust (`action`) a coefficient (`effect`) with a binary unmeasured confounder (`what`), we use the function `adjust_coef_with_binary()`. +Once you have estimated the above three quantities, we can calculate a updated effect estimate between the exposure and outcome that takes into account an unmeasured factor like the one specified. +We can use the {tipr} R package to perform these analyses. +The functions in the {tipr} package follow a unified grammar. +The function names follow this form: `{action}_{effect}_with_{what}`. +For example, to adjust (`action`) a coefficient (`effect`) with a binary unmeasured confounder (`what`), we use the function `adjust_coef_with_binary()`. Below is a copy of the table included in @lucy2022tipr about this package. -+----------+--------------------+----------------------------------------------+ -| category | Function term | Use | -+==========+====================+==============================================+ -|**action**| `adjust` | These functions adjust observed effects, | -| | | requiring both the unmeasured | -| | | confounder-exposure relationship and | -| | | unmeasured confounder-outcome relationship to| -| | | be specified. | -+----------+--------------------+----------------------------------------------+ -| | `tip` | These functions tip observed effects. Only | -| | | one relationship, either the unmeasured | -| | | confounder-exposure relationship or | -| | | unmeasured confounder-outcome relationship | -| | | needs to be specified. | -+----------+--------------------+----------------------------------------------+ -|**effect**| `coef` | These functions specify an observed | -| | | coefficient from a linear, log-linear, | -| | | logistic, or Cox proportional hazards model | -+----------+--------------------+----------------------------------------------+ -| | `rr` | These functions specify an observed | -| | | relative risk | -+----------+--------------------+----------------------------------------------+ -| | `or` | These functions specify an observed | -| | | odds ratio | -+----------+--------------------+----------------------------------------------+ -| | `hr` | These functions specify an observed | -| | | hazard ratio | -| | | -+----------+--------------------+----------------------------------------------+ -|**what** | `continuous` | These functions specify an unmeasured -| | | standardized Normally distributed confounder. -| | | These functions will include the parameters -| | | `exposure_confounder_effect` and -| | | `confounder_outcome_effect` -+----------+--------------------+----------------------------------------------+ -| | `binary` | These functions specify an unmeasured binary -| | | confounder. These functions will include the -| | | parameters `exposed_confounder_prev`, -| | | `unexposed_confounder_prev`, and -| | | `confounder_outcome_effect` -+----------+--------------------+----------------------------------------------+ -| | `r2` | These functions specify an unmeasured -| | | confounder parameterized by specifying the -| | | percent of variation in the exposure / outcome -| | | explained by the unmeasured confounder. These -| | | functions will include the parameters -| | | `confounder_exposure_r2` and -| | | `outcome_exposure_r2` -+----------+--------------------+----------------------------------------------+ +| category | Function term | Use | +|----------------|-----------------|---------------------------------------| +| **action** | `adjust` | These functions adjust observed effects, requiring both the unmeasured \| confounder-exposure relationship and unmeasured confounder-outcome relationship to be specified. | +| | `tip` | These functions tip observed effects. Only one relationship, either the unmeasured confounder-exposure relationship or unmeasured confounder-outcome relationship needs to be specified. | +| **effect** | `coef` | These functions specify an observed coefficient from a linear, log-linear, logistic, or Cox proportional hazards model | +| | `rr` | These functions specify an observed relative risk | +| | `or` | These functions specify an observed odds ratio | +| | `hr` | These functions specify an observed hazard ratio | +| **what** | `continuous` | These functions specify an unmeasured standardized Normally distributed confounder. These functions will include the parameters `exposure_confounder_effect` and `confounder_outcome_effect` | +| | `binary` | These functions specify an unmeasured binary confounder. These functions will include the parameters `exposed_confounder_prev`, `unexposed_confounder_prev`, and `confounder_outcome_effect` | +| | `r2` | These functions specify an unmeasured confounder parameterized by specifying the percent of variation in the exposure / outcom explained by the unmeasured confounder. These functions will include the parameters `confounder_exposure_r2` and `outcome_exposure_r2` | + : Grammar of `tipr` functions. {#tbl-sens} You can find full documentation here: [r-causal.github.io/tipr/](https://r-causal.github.io/tipr/) #### Example -Ok, now we have everything we need to perform our sensitivity analysis. @tbl-sens provides what we need; we can use the {tipr} R package to apply these; we will use the `adjust_coef` function. Let's plug in the quantities we established above for each of the three parameters, `effect_observed`, `exposure_confounder_effect`, and `confounder_outcome_effect`. +Ok, now we have everything we need to perform our sensitivity analysis. +@tbl-sens provides what we need; we can use the {tipr} R package to apply these; we will use the `adjust_coef` function. +Let's plug in the quantities we established above for each of the three parameters, `effect_observed`, `exposure_confounder_effect`, and `confounder_outcome_effect`. ```{r} library(tipr) @@ -950,9 +933,18 @@ adjust_coef( ) ``` -Examining this output, we see that if there were an unmeasured confounder like the one we specified above, our observed effect, 6.58, would be attenuated to 6.19. That is, rather than the effect of extra magic morning hours increasing the average posted wait time at 9am by 6.58 minutes, the true effect would be 6.19 minutes, assuming our specifications about the unmeasured confounder were correct. Take a look at @tbl-alt-sets, this number should look familiar. +Examining this output, we see that if there were an unmeasured confounder like the one we specified above, our observed effect, 6.58, would be attenuated to 6.19. +That is, rather than the effect of extra magic morning hours increasing the average posted wait time at 9am by 6.58 minutes, the true effect would be 6.19 minutes, assuming our specifications about the unmeasured confounder were correct. +Take a look at @tbl-alt-sets, this number should look familiar. -In this case, our "guesses" about the relationship between our unmeasured confounder and the exposure and outcome were accurate because it was, in fact, measured! In reality, we often have to use other techniques to come up with these guesses. Sometimes, we have access to different data or previous studies that would allow us to quantify these effects. Sometimes, we have some information about one of the effects, but not the other (i.e., we have a guess for the impact of the historic high temperature on the average posted wait time, but not on whether there are extra magic morning hours). In these cases, one solution is an *array*-based approach where we specify the effect we are sure of and vary the one we are not. Let's see an example of that. We can plot this array to help us see the impact of this potential confounder. Examining @fig-sens-array, we can see, for example, that if there was a one standard deviation difference in historic high temperature such that extra magic morning hours were 9 degrees cooler on average than days without extra magic morning hours, the true causal effect of extra magic morning hours on the average posted wait time at 9am would be 4.28 minutes, rather than the observed 6.58 minutes. +In this case, our "guesses" about the relationship between our unmeasured confounder and the exposure and outcome were accurate because it was, in fact, measured! +In reality, we often have to use other techniques to come up with these guesses. +Sometimes, we have access to different data or previous studies that would allow us to quantify these effects. +Sometimes, we have some information about one of the effects, but not the other (i.e., we have a guess for the impact of the historic high temperature on the average posted wait time, but not on whether there are extra magic morning hours). +In these cases, one solution is an *array*-based approach where we specify the effect we are sure of and vary the one we are not. +Let's see an example of that. +We can plot this array to help us see the impact of this potential confounder. +Examining @fig-sens-array, we can see, for example, that if there was a one standard deviation difference in historic high temperature such that extra magic morning hours were 9 degrees cooler on average than days without extra magic morning hours, the true causal effect of extra magic morning hours on the average posted wait time at 9am would be 4.28 minutes, rather than the observed 6.58 minutes. ```{r} #| label: fig-sens-array @@ -987,9 +979,11 @@ ggplot( ) ``` - -Many times, we find ourselves without any certainty about a potential unmeasured confounder's impact on the exposure and outcome. One approach could be to decide on a range of *both* values to examine. @fig-sens-array-2 is an example of this. One thing we can learn by looking at this graph is that the adjusted effect will cross the null when a one standard deviation change in the historic high temperature changes the average posed wait time by at least ~7 minutes and there is around a one standard deviation difference between the average historic temperature on extra magic morning days compared to those without. This is known as a *tipping point*. - +Many times, we find ourselves without any certainty about a potential unmeasured confounder's impact on the exposure and outcome. +One approach could be to decide on a range of *both* values to examine. +@fig-sens-array-2 is an example of this. +One thing we can learn by looking at this graph is that the adjusted effect will cross the null when a one standard deviation change in the historic high temperature changes the average posed wait time by at least \~7 minutes and there is around a one standard deviation difference between the average historic temperature on extra magic morning days compared to those without. +This is known as a *tipping point*. ```{r} #| label: fig-sens-array-2 @@ -1037,12 +1031,20 @@ ggplot( ) ``` - #### Tipping point analyses -Tipping point sensitivity analyses aim to determine the characteristics of an unmeasured confounder that would change the observed effect to a specific value, often the null. Instead of exploring a range of values for unknown sensitivity parameters, it identifies the value that would "tip" the observed effect. This approach can be applied to point estimates or confidence interval bounds. The analysis calculates the smallest possible effect of an unmeasured confounder that would cause this tipping. By rearranging equations and setting the adjusted outcome to the null (or any value of interest), we can solve for a single sensitivity parameter, given other parameters. The {tipr} R package also provides functions to perform these calculations for various scenarios, including different effect measures, confounder types, and known relationships. +Tipping point sensitivity analyses aim to determine the characteristics of an unmeasured confounder that would change the observed effect to a specific value, often the null. +Instead of exploring a range of values for unknown sensitivity parameters, it identifies the value that would "tip" the observed effect. +This approach can be applied to point estimates or confidence interval bounds. +The analysis calculates the smallest possible effect of an unmeasured confounder that would cause this tipping. +By rearranging equations and setting the adjusted outcome to the null (or any value of interest), we can solve for a single sensitivity parameter, given other parameters. +The {tipr} R package also provides functions to perform these calculations for various scenarios, including different effect measures, confounder types, and known relationships. -Using the example above, let's use the `tip_coef` function to see what would tip our observed coefficient. For this, we only need to specify one, either the exposure-unmeasured confounder effect *or* the unmeasured confounder-outcome effect and the function will calculate the other that would result in "tipping" the observed effect to the null. Let's first replicate what we saw in @fig-sens-array-2. Let's specify the unmeasured confounder effect-outcome effect of -7 minutes. The output below tells us that if there is an unmeasured confounder with an effect on the outcome of -7 minutes, it would need to have a diffence of -0.94 in order to tip our observed effect of 6.58 minutes. +Using the example above, let's use the `tip_coef` function to see what would tip our observed coefficient. +For this, we only need to specify one, either the exposure-unmeasured confounder effect *or* the unmeasured confounder-outcome effect and the function will calculate the other that would result in "tipping" the observed effect to the null. +Let's first replicate what we saw in @fig-sens-array-2. +Let's specify the unmeasured confounder effect-outcome effect of -7 minutes. +The output below tells us that if there is an unmeasured confounder with an effect on the outcome of -7 minutes, it would need to have a diffence of -0.94 in order to tip our observed effect of 6.58 minutes. ```{r} tip_coef( @@ -1051,7 +1053,8 @@ tip_coef( ) ``` -Instead, let's suppose we were pretty sure about the unmeasured confounder-outcome effect we assumed above, -2.3, but we weren't sure what to expect in terms of the relationship between historic high temperature and whether the day had extra magic morning hours. Let's see how big the difference there would have to be for a confounder like this to make our observed effect of 6.58 minutes tip to 0 minutes. +Instead, let's suppose we were pretty sure about the unmeasured confounder-outcome effect we assumed above, -2.3, but we weren't sure what to expect in terms of the relationship between historic high temperature and whether the day had extra magic morning hours. +Let's see how big the difference there would have to be for a confounder like this to make our observed effect of 6.58 minutes tip to 0 minutes. ```{r} tip_coef( @@ -1060,8 +1063,9 @@ tip_coef( ) ``` -This shows that we would need an effect of -2.86 between the expsoure and the confounder. In other words, for this particular example, we would need the average difference in historic temperature to be ~25 degrees (-2.86 times our standard deviation, 9), to change our effect to the null. This is a pretty huge, and potentially implausible, effect. If we were missing historic high temperature, and assuming we are correct about our -2.3 estimate, we could feel pretty confident that missing this variable is not skewing our result so much that we are directionally wrong. - - +This shows that we would need an effect of -2.86 between the expsoure and the confounder. +In other words, for this particular example, we would need the average difference in historic temperature to be \~25 degrees (-2.86 times our standard deviation, 9), to change our effect to the null. +This is a pretty huge, and potentially implausible, effect. +If we were missing historic high temperature, and assuming we are correct about our -2.3 estimate, we could feel pretty confident that missing this variable is not skewing our result so much that we are directionally wrong. ### Other types of QBA diff --git a/chapters/22-machine-learning.qmd b/chapters/22-machine-learning.qmd index be1d37d..25252bf 100644 --- a/chapters/22-machine-learning.qmd +++ b/chapters/22-machine-learning.qmd @@ -17,5 +17,3 @@ rnorm(5) ``` ## Targeted Learning - - diff --git a/chapters/24-evidence.qmd b/chapters/24-evidence.qmd index 9d3bca1..ff58d00 100644 --- a/chapters/24-evidence.qmd +++ b/chapters/24-evidence.qmd @@ -8,8 +8,7 @@ status("unstarted") ``` -> "Circumstantial evidence is a very tricky thing. It may seem to point very straight to one thing, but if you shift your own point of view a little, you may find it pointing in an equally uncompromising manner to something entirely different” -> --- Sherlock Holmes +> "Circumstantial evidence is a very tricky thing. It may seem to point very straight to one thing, but if you shift your own point of view a little, you may find it pointing in an equally uncompromising manner to something entirely different” --- Sherlock Holmes ## Triangulation From 3fe513a2d5ae4555d25d3c68059267cc9d95eff9 Mon Sep 17 00:00:00 2001 From: Malcolm Barrett Date: Mon, 14 Oct 2024 14:27:48 -0400 Subject: [PATCH 2/6] run sytler --- chapters/11-estimands.qmd | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/chapters/11-estimands.qmd b/chapters/11-estimands.qmd index 174b6e2..62a560b 100644 --- a/chapters/11-estimands.qmd +++ b/chapters/11-estimands.qmd @@ -971,9 +971,9 @@ In other words, the effect estimate of `exposure` on `outcome` should be the sam #| fig-cap: "A DAG showing the causal relationship between `outcome`, `exposure`, and `covariate`. `exposure` and `covariate` both cause `outcome`, but there is no relationship between `exposure` and `covariate`. In a logistic regression, the odds ratio for exposure will be non-collapsible over strata of covariate." library(ggdag) dagify( - outcome ~ exposure + covariate, + outcome ~ exposure + covariate, coords = time_ordered_coords() -) |> +) |> ggdag(use_text = FALSE) + geom_dag_text_repel(aes(label = name), box.padding = 1.8, direction = "x") + theme_dag() @@ -985,8 +985,8 @@ Let's simulate this. set.seed(123) n <- 10000 -exposure <- rbinom(n, 1, 0.5) -covariate <- rbinom(n, 1, 0.5) +exposure <- rbinom(n, 1, 0.5) +covariate <- rbinom(n, 1, 0.5) outcome <- rbinom(n, 1, plogis(-0.5 + exposure + 2 * covariate)) ``` @@ -1024,7 +1024,7 @@ We can calculate the odds ratio using this frequency table: ((`r marginal_table[ This odds ratio is the same result we get with logistic regression when we exponentiate the results. ```{r} -glm(outcome ~ exposure, family = binomial()) |> +glm(outcome ~ exposure, family = binomial()) |> broom::tidy(exponentiate = TRUE) ``` @@ -1032,7 +1032,7 @@ This is a little off from the simulation model coefficient of `exp(1)`. We get closer when we add in `covariate`. ```{r} -glm(outcome ~ exposure + covariate, family = binomial()) |> +glm(outcome ~ exposure + covariate, family = binomial()) |> broom::tidy(exponentiate = TRUE) ``` From 88c858ea06b481ad44b3f98f547bc1897c9ccc57 Mon Sep 17 00:00:00 2001 From: Malcolm Barrett Date: Mon, 14 Oct 2024 14:52:18 -0400 Subject: [PATCH 3/6] collapse chapters --- ...ps-models.qmd => 08-propensity-scores.qmd} | 102 ++++++++++++++++- chapters/09-using-ps.qmd | 105 ------------------ ...s-exposures.qmd => 13-other-exposures.qmd} | 31 +++++- chapters/14-categorical-exposures.qmd | 19 ---- 4 files changed, 128 insertions(+), 129 deletions(-) rename chapters/{08-building-ps-models.qmd => 08-propensity-scores.qmd} (78%) delete mode 100644 chapters/09-using-ps.qmd rename chapters/{13-continuous-exposures.qmd => 13-other-exposures.qmd} (94%) diff --git a/chapters/08-building-ps-models.qmd b/chapters/08-propensity-scores.qmd similarity index 78% rename from chapters/08-building-ps-models.qmd rename to chapters/08-propensity-scores.qmd index be4aa45..ed73833 100644 --- a/chapters/08-building-ps-models.qmd +++ b/chapters/08-propensity-scores.qmd @@ -1,4 +1,6 @@ -# Building propensity score models {#sec-building-models} +# Propensity scores + +## Building propensity score models {#sec-building-models} {{< include 00-setup.qmd >}} @@ -337,3 +339,101 @@ From a frequentist standpoint, the confidence intervals will also not have nomin In practice, cross-validation, a technique to reduce overfitting, is often used with causal models that use machine learning, as we'll discuss in [Chapter -@sec-causal-ml]. ::: + +## Using the propensity scores {#sec-using-ps} + +The propensity score is a *balancing* tool -- we use it to help us make our exposure groups *exchangeable*. +There are many ways to incorporate the propensity score into an analysis. +Commonly used techniques include stratification (estimating the causal effect within propensity score stratum), matching, weighting, and direct covariate adjustment. +In this section, we will focus on *matching* and *weighting*; other techniques will be discussed once we introduce the *outcome model*. +Recall at this point in the book we are still in the *design* phase. +We have not yet incorporated the outcome into our analysis at all. + +## Matching + +Ultimately, we want the exposed and unexposed observations to be *exchangeable* with respect to the confounders we have proposed in our DAG (so we can use the observed effect for one to estimate the counterfactual for the other). +One way to do this is to ensure that each observation in our analysis sample has at least one observation of the opposite exposure that has *match*ing values for each of these confounders. +If we had a small number of binary confounders, for example, we might be able to construct an *exact match* for observations (and only include those for whom such a match exists), but as the number and continuity of confounders increases, exact matching becomes less feasible. +This is where the propensity score, a summary measure of all of the confounders, comes in to play. + +Let's setup the data as we did in @sec-building-models. + +```{r} +library(broom) +library(touringplans) + +seven_dwarfs_9 <- seven_dwarfs_train_2018 |> filter(wait_hour == 9) +``` + +We can re-fit the propensity score using the `{MatchIt}` package, as below. +Notice here the `matchit` function fit a logistic regression model for our propensity score, as we had in @sec-building-models. +There were 60 days in 2018 where the Magic Kingdom had extra magic morning hours. +For each of these 60 exposed days, `matchit` found a comparable unexposed day, by implementing a nearest-neighbor match using the constructed propensity score. +Examining the output, we also see that the target estimand is an "ATT" (do not worry about this yet, we will discuss this and several other estimands in @sec-estimands). + +```{r} +library(MatchIt) +m <- matchit( + park_extra_magic_morning ~ park_ticket_season + park_close + park_temperature_high, + data = seven_dwarfs_9 +) +m +``` + +We can use the `get_matches` function to create a data frame with the original variables that only consists of those who were matched. +Notice here our sample size has been reduced from the original 354 days to 120. + +```{r} +matched_data <- get_matches(m) +glimpse(matched_data) +``` + +## Weighting + +One way to think about matching is as a crude "weight" where everyone who was matched gets a weight of 1 and everyone who was not matched gets a weight of 0 in the final sample. +Another option is to allow this weight to be smooth, applying a weight to allow, on average, the covariates of interest to be balanced in the weighted population. +To do this, we will construct a weight using the propensity score. +There are many different weights that can be applied, depending on your target estimand of interest (see @sec-estimands for details). +For this section, we will focus on the "Average Treatment Effect" weights, commonly referred to as an "inverse probability weight". +The weight is constructed as follows, where each observation is weighted by the *inverse* of the probability of receiving the exposure they received. + +$$w_{ATE} = \frac{X}{p} + \frac{(1 - X)}{1 - p}$$ + +For example, if observation 1 had a very high likelihood of being exposed given their pre-exposure covariates ($p = 0.9$), but they in fact were *not* exposed, their weight would be 10 ($w_1 = 1 / (1 - 0.9)$). +Likewise, if observation 2 had a very high likelihood of being exposed given their pre-exposure covariates ($p = 0.9$), and they *were* exposed, their weight would be 1.1 ($w_2 = 1 / 0.9$). +Intuitively, we give *more* weight to observations who, based on their measured confounders, appear to have useful information for constructing a counterfactual -- we would have predicted that they were exposed and but by chance they were not, or vice-versa. +The `{propensity}` package is useful for implementing propensity score weighting. + +```{r} +library(propensity) + +seven_dwarfs_9_with_ps <- + glm( + park_extra_magic_morning ~ park_ticket_season + park_close + park_temperature_high, + data = seven_dwarfs_9, + family = binomial() + ) |> + augment(type.predict = "response", data = seven_dwarfs_9) +seven_dwarfs_9_with_wt <- seven_dwarfs_9_with_ps |> + mutate(w_ate = wt_ate(.fitted, park_extra_magic_morning)) +``` + +@tbl-df-wt shows the weights in the first column. + +```{r} +#| label: tbl-df-wt +#| tbl-cap: > +#| The first six observations in the `seven_dwarfs_9_with_wt` dataset, including their propensity scores in the `.fitted` column and weight in the `w_ate` column. +seven_dwarfs_9_with_wt |> + select( + w_ate, + .fitted, + park_date, + park_extra_magic_morning, + park_ticket_season, + park_close, + park_temperature_high + ) |> + head() |> + knitr::kable() +``` diff --git a/chapters/09-using-ps.qmd b/chapters/09-using-ps.qmd deleted file mode 100644 index af74ad8..0000000 --- a/chapters/09-using-ps.qmd +++ /dev/null @@ -1,105 +0,0 @@ -# Using the propensity score {#sec-using-ps} - -{{< include 00-setup.qmd >}} - -```{r} -#| echo: false -# TODO: remove when first edition complete -status("polishing") -``` - -The propensity score is a *balancing* tool -- we use it to help us make our exposure groups *exchangeable*. -There are many ways to incorporate the propensity score into an analysis. -Commonly used techniques include stratification (estimating the causal effect within propensity score stratum), matching, weighting, and direct covariate adjustment. -In this section, we will focus on *matching* and *weighting*; other techniques will be discussed once we introduce the *outcome model*. -Recall at this point in the book we are still in the *design* phase. -We have not yet incorporated the outcome into our analysis at all. - -## Matching - -Ultimately, we want the exposed and unexposed observations to be *exchangeable* with respect to the confounders we have proposed in our DAG (so we can use the observed effect for one to estimate the counterfactual for the other). -One way to do this is to ensure that each observation in our analysis sample has at least one observation of the opposite exposure that has *match*ing values for each of these confounders. -If we had a small number of binary confounders, for example, we might be able to construct an *exact match* for observations (and only include those for whom such a match exists), but as the number and continuity of confounders increases, exact matching becomes less feasible. -This is where the propensity score, a summary measure of all of the confounders, comes in to play. - -Let's setup the data as we did in @sec-building-models. - -```{r} -library(broom) -library(touringplans) - -seven_dwarfs_9 <- seven_dwarfs_train_2018 |> filter(wait_hour == 9) -``` - -We can re-fit the propensity score using the `{MatchIt}` package, as below. -Notice here the `matchit` function fit a logistic regression model for our propensity score, as we had in @sec-building-models. -There were 60 days in 2018 where the Magic Kingdom had extra magic morning hours. -For each of these 60 exposed days, `matchit` found a comparable unexposed day, by implementing a nearest-neighbor match using the constructed propensity score. -Examining the output, we also see that the target estimand is an "ATT" (do not worry about this yet, we will discuss this and several other estimands in @sec-estimands). - -```{r} -library(MatchIt) -m <- matchit( - park_extra_magic_morning ~ park_ticket_season + park_close + park_temperature_high, - data = seven_dwarfs_9 -) -m -``` - -We can use the `get_matches` function to create a data frame with the original variables that only consists of those who were matched. -Notice here our sample size has been reduced from the original 354 days to 120. - -```{r} -matched_data <- get_matches(m) -glimpse(matched_data) -``` - -## Weighting - -One way to think about matching is as a crude "weight" where everyone who was matched gets a weight of 1 and everyone who was not matched gets a weight of 0 in the final sample. -Another option is to allow this weight to be smooth, applying a weight to allow, on average, the covariates of interest to be balanced in the weighted population. -To do this, we will construct a weight using the propensity score. -There are many different weights that can be applied, depending on your target estimand of interest (see @sec-estimands for details). -For this section, we will focus on the "Average Treatment Effect" weights, commonly referred to as an "inverse probability weight". -The weight is constructed as follows, where each observation is weighted by the *inverse* of the probability of receiving the exposure they received. - -$$w_{ATE} = \frac{X}{p} + \frac{(1 - X)}{1 - p}$$ - -For example, if observation 1 had a very high likelihood of being exposed given their pre-exposure covariates ($p = 0.9$), but they in fact were *not* exposed, their weight would be 10 ($w_1 = 1 / (1 - 0.9)$). -Likewise, if observation 2 had a very high likelihood of being exposed given their pre-exposure covariates ($p = 0.9$), and they *were* exposed, their weight would be 1.1 ($w_2 = 1 / 0.9$). -Intuitively, we give *more* weight to observations who, based on their measured confounders, appear to have useful information for constructing a counterfactual -- we would have predicted that they were exposed and but by chance they were not, or vice-versa. -The `{propensity}` package is useful for implementing propensity score weighting. - -```{r} -library(propensity) - -seven_dwarfs_9_with_ps <- - glm( - park_extra_magic_morning ~ park_ticket_season + park_close + park_temperature_high, - data = seven_dwarfs_9, - family = binomial() - ) |> - augment(type.predict = "response", data = seven_dwarfs_9) -seven_dwarfs_9_with_wt <- seven_dwarfs_9_with_ps |> - mutate(w_ate = wt_ate(.fitted, park_extra_magic_morning)) -``` - -@tbl-df-wt shows the weights in the first column. - -```{r} -#| label: tbl-df-wt -#| tbl-cap: > -#| The first six observations in the `seven_dwarfs_9_with_wt` dataset, including their propensity scores in the `.fitted` column and weight in the `w_ate` column. -seven_dwarfs_9_with_wt |> - select( - w_ate, - .fitted, - park_date, - park_extra_magic_morning, - park_ticket_season, - park_close, - park_temperature_high - ) |> - head() |> - knitr::kable() -``` diff --git a/chapters/13-continuous-exposures.qmd b/chapters/13-other-exposures.qmd similarity index 94% rename from chapters/13-continuous-exposures.qmd rename to chapters/13-other-exposures.qmd index 1a88ee2..5bb6bfe 100644 --- a/chapters/13-continuous-exposures.qmd +++ b/chapters/13-other-exposures.qmd @@ -1,4 +1,6 @@ -# Continuous exposures {#sec-continuous-exposures} +# Continuous and categorical exposures + +## Continuous exposures {#sec-continuous-exposures} {{< include 00-setup.qmd >}} @@ -8,7 +10,7 @@ status("wip") ``` -## Calculating propensity scores for continuous exposures +### Calculating propensity scores for continuous exposures Propensity scores generalize to many other types of exposures, including continuous exposures. At its heart, the workflow is the same: we fit a model where the exposure is the outcome and then use that model to weight a second outcome model. @@ -109,7 +111,7 @@ lm( ) ``` -## Diagnostics and stabilization +### Diagnostics and stabilization Continuous exposure weights, however, are very sensitive to modeling choices. One problem, in particular, is the existence of extreme weights, an issue that can also affect other types of exposures. @@ -291,4 +293,25 @@ wait_times_wts |> Our model predicted a much lower posted wait time than observed, so this date was upweighted. We don't know why the posted time was so high (the actual time was much lower), but we did find an artist rendering from that date of [Pluto digging for Seven Dwarfs Mine Train treasure](https://disneyparks.disney.go.com/blog/2018/06/disney-doodle-pluto-sniffs-out-fun-at-seven-dwarfs-mine-train/). -## Fitting the outcome model for continuous exposures +### Fitting the outcome model for continuous exposures + +## Categorical exposures + +{{< include 00-setup.qmd >}} + +```{r} +#| echo: false +# TODO: remove when first edition complete +status("unstarted") +``` + +## Calculating propensity scores for categorical exposures + +```{r} +rnorm(5) +``` + +### Diagnostics with many categories + +### Fitting the outcome model again + diff --git a/chapters/14-categorical-exposures.qmd b/chapters/14-categorical-exposures.qmd index a5ba685..e69de29 100644 --- a/chapters/14-categorical-exposures.qmd +++ b/chapters/14-categorical-exposures.qmd @@ -1,19 +0,0 @@ -# Categorical exposures - -{{< include 00-setup.qmd >}} - -```{r} -#| echo: false -# TODO: remove when first edition complete -status("unstarted") -``` - -## Calculating propensity scores for categorical exposures - -```{r} -rnorm(5) -``` - -## Diagnostics with many categories - -## Fitting the outcome model again From 7a9e64cd9252ea1fb788da7c96266bbdcb75ca16 Mon Sep 17 00:00:00 2001 From: Malcolm Barrett Date: Mon, 14 Oct 2024 14:52:55 -0400 Subject: [PATCH 4/6] move mediation --- chapters/{20-mediation.qmd => 18-mediation.qmd} | 0 chapters/{18-longitudinal.qmd => 19-longitudinal.qmd} | 0 chapters/{19-survival.qmd => 20-survival.qmd} | 0 3 files changed, 0 insertions(+), 0 deletions(-) rename chapters/{20-mediation.qmd => 18-mediation.qmd} (100%) rename chapters/{18-longitudinal.qmd => 19-longitudinal.qmd} (100%) rename chapters/{19-survival.qmd => 20-survival.qmd} (100%) diff --git a/chapters/20-mediation.qmd b/chapters/18-mediation.qmd similarity index 100% rename from chapters/20-mediation.qmd rename to chapters/18-mediation.qmd diff --git a/chapters/18-longitudinal.qmd b/chapters/19-longitudinal.qmd similarity index 100% rename from chapters/18-longitudinal.qmd rename to chapters/19-longitudinal.qmd diff --git a/chapters/19-survival.qmd b/chapters/20-survival.qmd similarity index 100% rename from chapters/19-survival.qmd rename to chapters/20-survival.qmd From 9d74aefba4c995f5a7b0bacf11b51aa4601f7a4a Mon Sep 17 00:00:00 2001 From: Malcolm Barrett Date: Mon, 14 Oct 2024 14:57:31 -0400 Subject: [PATCH 5/6] break out new chapters --- chapters/22-doubly-robust.qmd | 17 +++++++++++++++++ chapters/23-diff-in-diff.qmd | 17 +++++++++++++++++ 2 files changed, 34 insertions(+) create mode 100644 chapters/22-doubly-robust.qmd create mode 100644 chapters/23-diff-in-diff.qmd diff --git a/chapters/22-doubly-robust.qmd b/chapters/22-doubly-robust.qmd new file mode 100644 index 0000000..4a44e68 --- /dev/null +++ b/chapters/22-doubly-robust.qmd @@ -0,0 +1,17 @@ +# Doubly robust models {#sec-causal-ml} + +{{< include 00-setup.qmd >}} + +```{r} +#| echo: false +# TODO: remove when first edition complete +status("unstarted") +``` + +## Augmented propensity scores + +```{r} +rnorm(5) +``` + +## Targeted Learning diff --git a/chapters/23-diff-in-diff.qmd b/chapters/23-diff-in-diff.qmd new file mode 100644 index 0000000..73fdadf --- /dev/null +++ b/chapters/23-diff-in-diff.qmd @@ -0,0 +1,17 @@ +# Difference-in-difference {#sec-diff-in-diff} + +{{< include 00-setup.qmd >}} + +```{r} +#| echo: false +# TODO: remove when first edition complete +status("unstarted") +``` + +## Difference-in-Difference + +```{r} +rnorm(5) +``` + +## Synthetic controls From e8215d24d176aa0117608acccca1624511d6f546 Mon Sep 17 00:00:00 2001 From: Malcolm Barrett Date: Mon, 14 Oct 2024 15:06:12 -0400 Subject: [PATCH 6/6] rename files --- _quarto.yml | 50 +++++++++--------- ...evaluating-ps.qmd => 09-evaluating-ps.qmd} | 0 .../{11-estimands.qmd => 10-estimands.qmd} | 0 .../html/__packages | 0 ...s-2_b2c7b4df6c17916197a448bdcdd2e5b0.RData | Bin 0 -> 3337 bytes ...as-2_b2c7b4df6c17916197a448bdcdd2e5b0.rdb} | Bin 101224 -> 95210 bytes ...ias-2_b2c7b4df6c17916197a448bdcdd2e5b0.rdx | Bin 0 -> 179 bytes ...ias_7ad9b5764634ecaa2d807d51ea952c2f.RData | Bin 0 -> 3269 bytes ...bias_7ad9b5764634ecaa2d807d51ea952c2f.rdb} | Bin 52885 -> 47917 bytes ...-bias_7ad9b5764634ecaa2d807d51ea952c2f.rdx | Bin 0 -> 178 bytes .../figure-html/fig-finite-sample-bias-1.png | Bin .../fig-finite-sample-bias-2-1.png | Bin .../figure-html/fig-sd-ate-hist-1.png | Bin .../figure-html/fig-sd-atm-hist-1.png | Bin .../figure-html/fig-sd-ato-hist-1.png | Bin .../figure-html/fig-sd-att-hist-1.png | Bin .../figure-html/fig-sd-atu-hist-1.png | Bin .../figure-html/fig-sd-mirror-hist-ate-1.png | Bin .../figure-html/fig-sd-mirror-hist-atm-1.png | Bin .../figure-html/fig-sd-mirror-hist-ato-1.png | Bin .../figure-html/fig-sd-mirror-hist-att-1.png | Bin .../figure-html/fig-sd-mirror-hist-atu-1.png | Bin .../figure-html/unnamed-chunk-33-1.png | Bin .../figure-html/unnamed-chunk-42-1.png | Bin ...s-2_7d0b601f17d72438f53b6385725276eb.RData | Bin 3337 -> 0 bytes ...ias-2_7d0b601f17d72438f53b6385725276eb.rdx | Bin 257 -> 0 bytes ...ias_cc81dca70c2ff5532b8aeb5e3b3a3da1.RData | Bin 3270 -> 0 bytes ...-bias_cc81dca70c2ff5532b8aeb5e3b3a3da1.rdx | Bin 257 -> 0 bytes .../figure-html/unnamed-chunk-27-1.png | Bin 46276 -> 0 bytes .../figure-html/unnamed-chunk-34-1.png | Bin 76825 -> 0 bytes ...outcome-model.qmd => 11-outcome-model.qmd} | 0 ...r-exposures.qmd => 12-other-exposures.qmd} | 0 chapters/{15-g-comp.qmd => 13-g-comp.qmd} | 0 chapters/14-categorical-exposures.qmd | 0 ...{16-interaction.qmd => 14-interaction.qmd} | 0 ...qmd => 15-missingness-and-measurement.qmd} | 0 .../{18-mediation.qmd => 16-mediation.qmd} | 0 ...9-longitudinal.qmd => 17-longitudinal.qmd} | 0 .../{20-survival.qmd => 18-time-to-event.qmd} | 6 +-- ...{21-sensitivity.qmd => 19-sensitivity.qmd} | 0 ...doubly-robust.qmd => 20-doubly-robust.qmd} | 0 ...e-learning.qmd => 21-machine-learning.qmd} | 0 ...-and-friends.qmd => 22-iv-and-friends.qmd} | 0 43 files changed, 28 insertions(+), 28 deletions(-) rename chapters/{10-evaluating-ps.qmd => 09-evaluating-ps.qmd} (100%) rename chapters/{11-estimands.qmd => 10-estimands.qmd} (100%) rename chapters/{11-estimands_cache => 10-estimands_cache}/html/__packages (100%) create mode 100644 chapters/10-estimands_cache/html/sim-finite-sample-bias-2_b2c7b4df6c17916197a448bdcdd2e5b0.RData rename chapters/{11-estimands_cache/html/sim-finite-sample-bias-2_7d0b601f17d72438f53b6385725276eb.rdb => 10-estimands_cache/html/sim-finite-sample-bias-2_b2c7b4df6c17916197a448bdcdd2e5b0.rdb} (92%) create mode 100644 chapters/10-estimands_cache/html/sim-finite-sample-bias-2_b2c7b4df6c17916197a448bdcdd2e5b0.rdx create mode 100644 chapters/10-estimands_cache/html/sim-finite-sample-bias_7ad9b5764634ecaa2d807d51ea952c2f.RData rename chapters/{11-estimands_cache/html/sim-finite-sample-bias_cc81dca70c2ff5532b8aeb5e3b3a3da1.rdb => 10-estimands_cache/html/sim-finite-sample-bias_7ad9b5764634ecaa2d807d51ea952c2f.rdb} (88%) create mode 100644 chapters/10-estimands_cache/html/sim-finite-sample-bias_7ad9b5764634ecaa2d807d51ea952c2f.rdx rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-finite-sample-bias-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-finite-sample-bias-2-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-ate-hist-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-atm-hist-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-ato-hist-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-att-hist-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-atu-hist-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-mirror-hist-ate-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-mirror-hist-atm-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-mirror-hist-ato-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-mirror-hist-att-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/fig-sd-mirror-hist-atu-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/unnamed-chunk-33-1.png (100%) rename chapters/{11-estimands_files => 10-estimands_files}/figure-html/unnamed-chunk-42-1.png (100%) delete mode 100644 chapters/11-estimands_cache/html/sim-finite-sample-bias-2_7d0b601f17d72438f53b6385725276eb.RData delete mode 100644 chapters/11-estimands_cache/html/sim-finite-sample-bias-2_7d0b601f17d72438f53b6385725276eb.rdx delete mode 100644 chapters/11-estimands_cache/html/sim-finite-sample-bias_cc81dca70c2ff5532b8aeb5e3b3a3da1.RData delete mode 100644 chapters/11-estimands_cache/html/sim-finite-sample-bias_cc81dca70c2ff5532b8aeb5e3b3a3da1.rdx delete mode 100644 chapters/11-estimands_files/figure-html/unnamed-chunk-27-1.png delete mode 100644 chapters/11-estimands_files/figure-html/unnamed-chunk-34-1.png rename chapters/{12-outcome-model.qmd => 11-outcome-model.qmd} (100%) rename chapters/{13-other-exposures.qmd => 12-other-exposures.qmd} (100%) rename chapters/{15-g-comp.qmd => 13-g-comp.qmd} (100%) delete mode 100644 chapters/14-categorical-exposures.qmd rename chapters/{16-interaction.qmd => 14-interaction.qmd} (100%) rename chapters/{17-missingness-and-measurement.qmd => 15-missingness-and-measurement.qmd} (100%) rename chapters/{18-mediation.qmd => 16-mediation.qmd} (100%) rename chapters/{19-longitudinal.qmd => 17-longitudinal.qmd} (100%) rename chapters/{20-survival.qmd => 18-time-to-event.qmd} (54%) rename chapters/{21-sensitivity.qmd => 19-sensitivity.qmd} (100%) rename chapters/{22-doubly-robust.qmd => 20-doubly-robust.qmd} (100%) rename chapters/{22-machine-learning.qmd => 21-machine-learning.qmd} (100%) rename chapters/{23-iv-and-friends.qmd => 22-iv-and-friends.qmd} (100%) diff --git a/_quarto.yml b/_quarto.yml index 3d9cca8..52169a1 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -28,40 +28,40 @@ book: repo-actions: [edit, issue] chapters: - index.qmd - + - part: Asking Causal Questions - chapters: + chapters: - chapters/01-casual-to-causal.qmd - chapters/02-whole-game.qmd - chapters/03-counterfactuals.qmd - chapters/04-target-trials-std-methods.qmd - chapters/05-dags.qmd - chapters/06-not-just-a-stats-problem.qmd - - - part: The Design Phase - chapters: + + - part: The Design Phase + chapters: - chapters/07-prep-data.qmd - - chapters/08-building-ps-models.qmd - - chapters/09-using-ps.qmd - - chapters/10-evaluating-ps.qmd - - - part: Estimating Causal Effects - chapters: - - chapters/11-estimands.qmd - - chapters/12-outcome-model.qmd - - chapters/13-continuous-exposures.qmd - - chapters/14-categorical-exposures.qmd - - chapters/15-g-comp.qmd - - chapters/16-interaction.qmd - - chapters/17-missingness-and-measurement.qmd - - chapters/18-longitudinal.qmd - - chapters/19-survival.qmd - - chapters/20-mediation.qmd - - chapters/21-sensitivity.qmd - - chapters/22-machine-learning.qmd - - chapters/23-iv-and-friends.qmd + - chapters/08-propensity-scores.qmd + - chapters/09-evaluating-ps.qmd + + - part: Estimating Causal Effects + chapters: + - chapters/10-estimands.qmd + - chapters/11-outcome-model.qmd + - chapters/12-other-exposures.qmd + - chapters/13-g-comp.qmd + - chapters/14-interaction.qmd + - chapters/15-missingness-and-measurement.qmd + - chapters/16-mediation.qmd + - chapters/17-longitudinal.qmd + - chapters/18-time-to-event.qmd + - chapters/19-sensitivity.qmd + - chapters/20-doubly-robust.qmd + - chapters/21-machine-learning.qmd + - chapters/22-iv-and-friends.qmd + - chapters/23-diff-in-diff.qmd - chapters/24-evidence.qmd - + - chapters/99-references.qmd appendices: - appendices/A-bootstrap.qmd diff --git a/chapters/10-evaluating-ps.qmd b/chapters/09-evaluating-ps.qmd similarity index 100% rename from chapters/10-evaluating-ps.qmd rename to chapters/09-evaluating-ps.qmd diff --git a/chapters/11-estimands.qmd b/chapters/10-estimands.qmd similarity index 100% rename from chapters/11-estimands.qmd rename to chapters/10-estimands.qmd diff --git a/chapters/11-estimands_cache/html/__packages b/chapters/10-estimands_cache/html/__packages similarity index 100% rename from chapters/11-estimands_cache/html/__packages rename to chapters/10-estimands_cache/html/__packages diff --git a/chapters/10-estimands_cache/html/sim-finite-sample-bias-2_b2c7b4df6c17916197a448bdcdd2e5b0.RData b/chapters/10-estimands_cache/html/sim-finite-sample-bias-2_b2c7b4df6c17916197a448bdcdd2e5b0.RData new file mode 100644 index 0000000000000000000000000000000000000000..44390ce30d2ce31db56835690a83eae1dad0d947 GIT binary patch literal 3337 zcmV+k4fgUMiwFP!000001C3XCIFxG}AA82WOoZ~>I7e$I9Sqc@EEjsnRmeV<%bH3~P{(0Z~xwqf_{GR)MuJ?7> z<)I7pfI=V;J_tWQ4}?zu6d?i!+-#AC5C}gnxZr{CLxjP%D9Qy(CD14+CXq-0*J2O| zF9QOB=Yz7(ZbR4;-b`;hnDLRUkk4KWg@T+pB+%X0^qMRGIH^x)7;56(KN_<^`jEu4 zrI{A{)Uf#1RuiK=A}m`+0e)SbV$0)@L)tD6IVZnoxKf|UroZsZPujYpYTMvJX|d1& zCV{VVa*=DSxY+yT_PlN_B3HQBNAsJVOs95jFmAw=y$kZb**)Yv|3U#?VfWVzQ>WRF zsF!|4Z<~X+T{i8p$zl7WeVd=2ma&h47?oxdcrO*i`n;-3iMVQ@nl2dkA^@v`9H@7Q z&TiWH0RUpUo379QI>4tSaE{Z))xg_gYcx^#oIV@a#{D9qd^V{?dcB*|%nBs(_C(dh zh~{>ewan*~mO9q%`F$j>t1d1Zo+mCHrg(LXtg6(twaJRYB{sNRxLOxk#6ujyEJCqI zJJ?%ie%-w{Ho?E}^Oum89J#~*yBvk6KtDsmqk<||^vlT>t#M>GHS?NE=XS+Rl|fmP zyJUNNH5*`1JAzv)^3qNFdPA;D z=})#@xgQ;tSJx9Yrf;1F6F)Dyt!=E0_I_xGdc@d)>f|l4y!*9()jbCcotUhIe%ib- z%qVU!QKBtED^I!LGT!Qp|MYm{8#nWgpwZeyKGA>?iL&}VM=?7KHBiD$li7Rrx2wx^ zERN27*vG6QI9LwTHrjM_le*gA4E!z6Lzwz|O*g+B_dIla;l!|$3Gi-c-x%#?1M5+u z$G+Axp(HaN``e4V#-%Ld7n4J6u_8tng^r$_5K77!D!JvKEn>?KReR>rX`ks2wl$_6h*%!dR zmKzpT6h3s0?7kTL`9Pw2Y>;r{z-Zs?lXs)_i@#GyoXZ15<4QVNtX17&OO46*H)m}k zY}H(>w1%_y%2-c!;T5kR{I>n$h?h_hOn`D#rb5-@8EhFwmhyYoa4oHjmZ3#1~#=IYp^m`C|i9&DMsui~?@j(5xE zHy(TrOA;yVeIy+hzZH>vpjR1N9N5@piKkl&O3crGOL(ww%`-YX)GIX~7z#%zM|5px zS1VnPR#rKAc-R{27Es4jx3{fjC$!V(LX#ZcZ+B*0N@v4o2XU^w5vbRf`pr<6LblD| zF$b=}s?Ofn;I>a__&~db?NA>7)e(i5Ler!O5= zlGYJ5#}wg{F+Gd^-j}^t^OzYrM>m+L?bwxBy*Krr*xXS zf+b`>S=3{(*GOi^T(Q@7_Bt;w(dKT-4zxcO&FbLl+Y%y+gWl-^M=0#iJh`^fotF;g*9E!6IGu{N)%Qc<&WM47AQogkI7 zQhDN)Q=vV{%7?Xl+)HF4>;YukeD>%x-giT!Mk)7cPPV`e_VQ(6BJ9a5hF|ju>S&{ie z(w_aMwl4D|nPKmyEhwiLa+BVXihK{xT9L6ZJ5-vt>zs~j6xK;6I9@*Ax~;xS!17ap zP2zUayCh_%jLI!T*hF|eVoYf!a_ie&JRRqv{T`PfS?*tb6h>tM!S8lMLBqMNhwuH4 zsPOV{F6uGfe`xPR)#l7A(Ea_gz31XfG=fT;CG&;#Pr)`HdVHbEE_-iDqx)p-O*2Sl zM<`ySFvtFwh6BF->0a~8Ng3S13(^}S4r4as#pQP~=k%$S6AgP2C9j^$l|$@4*o!xq zeAN;$KYc2VQp%}rU3fazky|_~E%bfxQP_hzv3%U4yCDIZ-8xh5VwQ?0y)IVcyCZXhyVQ-B-Ci_gIY#V}8<211)VAC;$46QT_LbkGz8)H0a?j$*Li$r{T22M9cY2ag zr*x&_P|Uavbu{C8s&_z#DuP8D>b@1vH9ly>%4fg04@yX7AEro6B~> zRV_QmOPLjkAA6V|CaE*YiY=?ti^hCZ?DE9UXCKCi!lVV9wyTB?-^(N#hWG>NS^7|96C>}(9!N6JH(O^RO<|LdM9$4#*ui*IblZ4x!E?>AJcB_tDh5-jKhnQ`^U zGDQn61V!Vfhx&1C#~t`Q(adNWjt9>{U(~(gm$63q2YE^pVxmMky*6Aiy@cr-K;`jc z#;~V$!|4-7IH5cHZ=Dnvp6;70YaM*p=@fos(c7=tqMUi#KN}~PE&!L?_v^P8Td+5l z3Vw;VmOMR;N|rp7=d_d=<FnxAXy5({2Y-X153GGT1FNw}yc6ZD^;6}^7GWW47 zGHJ(YqEf!(qE5Vq=p;2t@*SaHHBagox7s<=Y6ctpYO>csKp}W{#>RP`FKIKys?@?5Kh1O_+uoSv03K4hJKWRFCUD2^te{A4&$A-}z6IS~#pWTMT(~~4JOnYF z9Dyp}+>xh}F5EJ7^A43z^mV7!1%(G47O0*g9@YPJ7%@`0#kHLGY0={Ho!@v0d9*Wm zhYJD3aTf`ia`>4CT$f9Bj&p+C^uW3WM2OYGo1VXY>$#wG&`2VaMW%qCd`xdV7Vk&Y z_G3{3wV7lJl0>GGSwtigOQ8o6kvKAziNtv0Fn9x;9)Yw2uVY}OvqQ(o0IR2Gh$G+$ z1PoChhX%7s{)xb!Vr*;-gro4pz`$crsE?0NIEn$RE|GW|fp`oG0s*E-fD}x{v&b|m zoT>?gLjgcd4Zt&qSQZh$0-#H9ASe<50+xkk5?P>qtv2tq+Ex;<$T(ae5e}LFAPg`A z7*raA0%k+3YS^F#4~BY&`VpyXP5^*L=>uAzv(^u1z#KpWk$?_j-NVi0fDO1tuP)b9 zalll6jRBRKe=o>8l*L>r$~TY#U&oIG!q(zf_5h~%$$!m4!cxeAE9r4$Dvd(M2EtY8 z3>uEiQe6cBBFtCau))3*B9#TF5<~wr7r^4si6}aQNFd``pjDMYWYTF=CQ>co_z?lu0gzxJ|BU;m0r7Wa+P@;x z{)tTce~@Ya3z_yOO@D1wDu_;9t7(-zuub7Bg8HA#Ab_70wDmoN63M=PtknorOE(); z1OUDq*LnpC4jVW^Hf*q^s6;>?Adtau2G};h`p7WBQ(z)~%oacdXCKf(qtRe_6e1G} zj@jxTAY4Zi`tx~$A*-iM!P316q?K(ji1fAH;Nkjc#7Y(fppRaY*2Hxgyn}m Tyo$h^>Dzw+MCUP43=aSR76^VV literal 0 HcmV?d00001 diff --git a/chapters/11-estimands_cache/html/sim-finite-sample-bias-2_7d0b601f17d72438f53b6385725276eb.rdb b/chapters/10-estimands_cache/html/sim-finite-sample-bias-2_b2c7b4df6c17916197a448bdcdd2e5b0.rdb similarity index 92% rename from chapters/11-estimands_cache/html/sim-finite-sample-bias-2_7d0b601f17d72438f53b6385725276eb.rdb rename to chapters/10-estimands_cache/html/sim-finite-sample-bias-2_b2c7b4df6c17916197a448bdcdd2e5b0.rdb index 34888637fa26e31c116cee7d5b92b2a2093f6c6f..60df806aa93e1cbcf640a2f9cd53b2c6d8659cec 100644 GIT binary patch delta 1658 zcmV-=28H?PmImtQ1%R{xh;4r-x_F%JSzAvVM-(38^%Wc2fH8M4_i#<3KD1KRs1I!v z^&xpkrM@8p3t0&pgcm`ARPj^$6DrZSs&RMlH{P=+GrP7iNglG&;dpk=eCM3+%^2r0X0UbgVnke88|F!?46f;{Y4GO_h1G1MCT6lJibH^{XZyKB0Mf#FFVTP4rSOJ3g zxQ;IoytFni{x;f3+7iJyvOK3O{KUHSPknRy|j*CN{oWG zj%seL-DR~{U-GzCcTQCa^M>Hk2TtRx?$k~hrA3N#esOl-xP~HiSZUZ-A>riQ_viNR z`GtMt9J(4O=?j0md4ASt+9r6fn)~e<$$UmkmArEIgOv1xXPqfXV_D2nbTyLI5y`1~#zBFr=|>-E}&^MMgF!mE_EkB{v`mk>Q_fQ;=h z-(x1jJ!Uv=FpPUaGSjjNbCVfP1PzxMY5A~Xko%Re$G3muCxMo7)lKrzi@TV=v1_PW68ZPMHx&Aw@*Og)-kH#<`LCS(KCr$1z8}2_i zr*0<#wf}UXs;ARpNOZElD#EAGQUqJeA#BP1m5_eX{Susxs_JS)oNM86@Fl4Ar}MiW z!QVziTu*-@_}lD(Khx}#I9uY;S6*a|wj;RM3CZ;{ue{LuM{yx@*$u{)Kx}H08$)}x z1^Yc=c!VA@;HM_Tm0$+5jE#BS%94Q3f`DtSaGVtrbN;@^c23^8Kz|-YU*$ETo4xqY zZ+{8zey#DXbPO!twSV9566F3?ekSmP3F%_3LtuaVbMvl~NBh%6jkhp%Cd;j8UwfP2X4CQZy002X_BE1HplN`HT++!#t4rJ%J|9$) zSp!Mx6o-sK|H^>8c%p-40 zhK!T;@}n+`@$#+sgC-8OLr{1-$U)+bF#H0+9|o=*{~N(I;Zd9t!6_onQP?W}$5EeA zXzWC*sK44}b>{Je@YnAhY9Vz6dJ=jJdK&rhk965L2R#ct13eGD06iuAx%1}MkDZI! zRa%gQ@c&gNB?X~l(i&0Y)wm3V+GKwr!x~X>sNKE7%R>|&3cmoM7)6bk@Dxr7LdT3k z$kpq!V4gl>zpc5!t+_?$B@w>nR)qgdmhENeRp>S7D)fdhuemMgP3R}k+t54E>*Cv* z`+{q31#@6l)w~ET=U6Xa_q$?n?_Gtf{FjCQED(reTKIEZu9;WDpC46rU(Ie4wO&+g z>PvX9-CpHfJQe!KJo>w9GwPVcMrTGyj*!x{c%kgW`mfNg^K-CA5wU`KIJae!QH-k4~{U3+*}Xk@ zsCWvrFEbs}H|;#pZJg+3aJ#JEGcRXgV@AbK@TIV2jrE18sh1k#5aE`OFC{${wna7V zVZ}T=0z;97iEdu0lmCeAW|h73S?R*xM?WGxs`|WoYS5zOyJJb>PUGeG-{U>w zZfGvt{zPa6S%nY2IS7;v7ZT<3p)$qGFFuFT`!(I4?P%<=$kZX*&uxUiFIpd(=v+Aa zr|#%{OPa1nRxc0I_*cwhZ;`p@E(4Db+#L+G&A4W%JCGZG7t7o28#-`MR_be-Y(&23 z(+k#x-41PV5e3^f_5w+VivGM1*K!dnCA7o71q5&LuF2^Bevn!YG$*P*Qhp+Md+}^> z&@l#t9MhX2zn45Kb^Yu>P~!D)1w$_(syuz^WM;d0;OV@t|1P{ctM^#xS`h<(Ufxyg z5g||^xCVu+`g0-vTJIZ)!a)?K>~3$P(tUBk&5G4vVw!l7GkXMv(kIC#*UuU$dsD8wg?ZM;Jv6FgLWtG z@g^$YF;l=P7AOgSv3XFLZ7_1~Xk!;;5il6JVKDObV{u1&-}`OgjCfxEb$yW@X(g!C zxz3&-b&H9>N|njxrhzo)^4}-+p$t<9zX_72R{ZT3)FyIc?N9lNcE~r?c)c_OYuk|U z_S%Faa)pUqhk32%)f--)`GVl@+w$sq^P~5;T;^59m1#D!gwNRh$?5pv-1|hQZ&}RU zuVoV1;m7h1DL0gg0FBBD3Q~EHW=a||r2 zI_muKI@A%rs(6LOyA>uKNf*@(m0_lGAs)yj8qX zRDDvkd|n0w;a&_ls2IPWhkbQAC3ySLH(N!+Y)^+bE&-a!^}xI1hm;LcMb!B1^WQu( zpS)bsuyE`^S>ekjI|mElvW7%OCz+Fl(qHa8-eLb>-NHw06Vk>;4Al%2NjIa}H6GBiWp7 z?dWY{T+2`h*apDaQkTd_X17P6YaNacgSzQi0gqOfLzB1PfbSpcJB{yus-L=e7-Kq) z{`Fa8>^9S=9kFN;&`YEVQ_0Vkr0g3YJ{WhHq_io?)e@06*HTR^Ydqu5IblB1(_k1p z3_@0I$r>RqNfOL-|I`bK5M16B(tNwjzrh>XiV2$4b_m5B~FCp}lNzIuwU#otoE9T(3O?ift81+JU z?`s=&`nSM4pS+@rlHqkhV@O)mipcmYHp1MaQM*${0*3JJ`aKG@JQQ0naWqwNz1fv{ zUziA(?MrlpRukzkpHV`-7@`pXlc^?TiEWdg@R-g)Z{Hzg(Sk+@ zx(Vg6Khf(jKm7hXq#$K9;W(!Vmg60{qNB<(GXF`CgykfMF@6We%1{~;ZUZ?J!NQ`f z;2#(p6g8qH+lb&rtVQT~DK>KCVDR_A=6AUbd#a4sYYrnBYK5Yi!dj&jDKYNwjj+6g zT!^4Hy`F8a7R9m-=2lZ$b;7 z=K*n!wL`2IFk*)%{I?wfzqu_Ljr{5X;ISN6*iVv<#|Z1;E;BTHB30US&RG&#g<|!k z4V}ZuQa!|B%#767DK2)hiv#)nr!3jc1`k)CkDnc^?FMQlintG8PJ(m9C~(P77ObcQ zPiV6%@~sWPG}cE4Kd;45$7nYMIqy#&O891mx1zZ1B*Q70Gs}9tLvMLF@+V@ghZoE z%h?CaN60R?A)lSwoO5%+QaDI1CAN?jYMw?Wb9?K%IcLVD`1|{1A~M} zY1)&MOSenHw6zA)~$12kB)R-lckP}!H}`qm(a8!F+fpyj*~c{KLWKu z(J48+(Vd*6H$WKB32~tMIHQ;tSGK$Y<{@H!+k|43a2x{{hraHe-xh@tuL1(f-OEZ{ zNG`Y>IJfBBWd)qf2-E__j@9P|*ZKe8sS|sn+~5@YI2Ubz_yeMdQM$|O|D~BNM&V4` z`9uG_(wE1=A#ejQlunnSGOyNsHUp-V(l~@JWI0>jsu0?5#8GOm(BkGMOEKWP>cYgN zvdmfcq|V`QaiZ-u|6?7;oId(nC!Wb(0lJWXPZ1e()^Ohz#reO7-fZ0(yk(pmb7-sw zo4jm#Xmws{uJ72L*xZZ0&aC_Sn0s}axD{PUY5Mky1aGKEcfhV8@PjrkmQQnhxOYZ> zoJ-z+=W2w+q21BDruPyv(Ru|CkwV0`tmfgAAzu}T+RPkQ8#BS&=U$ZZ(HsLsM=Vxu>$}rN95|mJh z?w68bkxq8fdI4f^qZOL=q7&}&Id`s)s|{B;stJ{x6W`EO6!hjD4E?SHM@Jd|lJu$Q zdr?icW((Y^Ao@r zh;8+9H;*uJp0L8aX0UgcIxY$0{IWAyqMTMcM_BY~TfYK;l$VBX!dTL#zsV|iAXzLr zN;-sC#=gaEqG?!;Ph_Xe;B*CBntK^MH+kqBIHt4iGGMIS9jLR|6Wss^bitc88MZ`A zO1to~q8o>vkP9%9iPe!pi$qWo8P_DP!tt+$tPZnzQ-}exqTbyRaun7@ph=QM|82U^ zUj&atzGvShvR9XhZ=-?w)09~GBJ5PNIMlv!S#!^u%NRO>8jf2XftsP{h9~uIaW!QX zwQR~iFDWf|g*(~O$R!ivTseJ;;v&ZEfOU|#w5N&V6_LWMRo@iQ+_WA|LjlU4Gs9Gg{ zQV7SHIlXJhr3X8lE5g0*-201QWH*49+rSDiQSTiT$7xPj$b}G*T;v9bb60_B*q+x8 z3do{Km@n)p;#^5Os-3L$Rv5W{6QxragWm5u%Mg40fZOyZJ8U z62(EAXb$lFR1QPUw$9?QhQ&L?{7jyPvsYK|G~M9-}k&_oPiR z5KIMgOyV`0sDih~Wy}FKYXVfPnkAbk#VF?D2tA4S$0g2jvaN=eJIF4`+5}Gm=e{iF z>JQH2ZbG^k3*hK4-g|@`HLxS1KXJ^sxh#zX5<^6n8%S2p1e)uYP;@E?m!loMz_~G( zDn(_UX9UyLU=Vk}kXu;TV?_#JZ-QL)vZ4&^3F}n-=b<>fx*B5 z6*d;Zkck|X3l0ZHE;!1kPiy#=Z`JL}0VX@Yb{KQ=!kA@7Yy+mpOZMKEKSygZZ~3sa z7aqu)OKO7`sZKxG>tFMCzl~xXDz`hb3tR7U{acPC#-?&v(s{$dKQZws{>ctK!5hH- z_yIOmiag#Y@=dOjJ~eW+yRXI-$WuWA*Y0Xkce=S9iUr}##Tl3?^vL972^(lksKhL+ zrgK|^U>5Z-{E+&sD~MlwpRK8Z;wpIDJ1*Sm-}Ea9?K0580RPne`a|N-v3C(Kxa-=T zftoO69wn#s`@Q=B0+eO- z;1U|6D_YBBZtocqq&yaZy@?!VfI7OCb>y~IUHv(=Jz78L{V~FnsDxvz@RHhak@MhkRLs9199sI|_U-=BuDtSU?1P9Oe$6Tb~#Lj$miveE@aMXA7XCpxeX?DYc zIX^(lZDm|?c8-8^U{}sq80UX&+o)qPTz_7-E4JT1x^A92?ePPedMK~(JT*&aZs=9g zfQV*}Q!x2eaon_EG-JQkQD2|GY`XC5`uOXMGyXnfcRoIXiQk;~e8};0(ZzA?W&Ygp zGp%~rdh4p~lpH{T0csEOSo{134#(fcGmeI1r`Ro8Qr`bY zv_JznreC~Y)~{K=zha1;0%sv8tM{xgYVW^R6|vd+-k|%rTfzGpQd{%UUj35&?~D4( zNN>VdK#M^fF}q0FOO@7O>3QFX6Y@c zn&jNvQ&^j6#r8Gj6zV!x^%czjqWp*|o-bfpnOSRvYAu}t9L4g}GFEKhF{#MVV-J%m zte?V$aV{)iaymgY#2EyhdaE>62Zb766c_4IX`J;I0*ASQ73)mpGZNK6=UOWt7#N*M zw>o&b12k^~idzrtdL17h0-iCt9dS@8eC26>@-Qwrvu)M&eK^zOe)rQG?C&iWS)8^V zYUXp;CHcGX^_`x3PIirkC4W^b@IT47_l%ur3JPYx^ixnNr`lrL?D`csTj{E_wLNaTop40E8bDSpyV@- zb}^NKEz)U~?D-p|*_jOrwhMCDU%hVshL^TA+&p|l46RpReuEEt`kq2YcK}+L%FFld(To4vuJ_+2(U5_4^~M{95FBB;rt z_#N;O{q-;W;+09S7y}K3qGuhR>r(?B#Vxl~vJ2+X{*9TN->HoX>+9={xD{J|fxgVA zxUsLcbN$cGZ>rZ%U~b>pa)h2Z0zMg#p3ozBR8&w`Iv!@I8 zQM+%Fy4%TLbd7*=m-1K@`_A(nbC^40V|*6C_tKc(?i^B2^PkKkK@%;q%QhlloIS?@G$F)8|GEWIwog>0{C|Z4EY~Ozm)}_>ir>7yYkFXYE+xx2E!p z6^nDze1jDe<(J1ff@>Uu6FfW?i2wdedBG>G#r(aW2lzGJcK|QAf&=K!&<&zNmoEY< z7IOSiwvWp%5s&;Tm*9OFVEb=m%HE^T(1W0RCk@- zUT8j;B@=B^JM)5+cb&y7`1O?U{s<=$plx1W5d#})HhYhu8`Ag~N4Buv9cz35c3k&c zRigJ7AxP7yJ7158`8HTK2jopYiRLG{RHDs`_++$ZlghP|Er_yh3m@k%m%kd~A5nV+ z7C5fn)+?5F=X+87dBNDFS9l((Ra+a_fuFCZ;qb?Mm!VJhB}Vj1{CT|M@IEgQKtoYg zV;tP`P8~NYHx83Mq*2*r>_;WXM)Q;U^a#p|*ibiU}qEPAmtHT$SEj7Vz<6 zOr`(!=3z>;1H!v41@A^>lOF1YP^v?2F*SUwXqjq+ME5f2>x}R0-!Zp_ZqIBAod8L6 zWh&WQW;c~-j7E=M%%((lUo zwb1dM$PDcz(=sD8SR?Gik4& zU36wX7SXHZDU%8n#Bu@0^@8PZ09Np63!z7`^=#@p;B5PmW zuZYr|Ov7BGgRiJR-+MI`{0MrbeMNn|I`H_H=TD|qLKPne~ ze!ilA7O`?_pn17_ChDg_ZE>yy2q<+d4V$TKJ|gv5Y5-(xjWD@UZR?*~qU!K$#sAIS z{O+pGX|SH$wTDyVw>pGBDyLShK<*xU2pTYQ2xoopmfnsrZ~td3+OOtC!DvL1hq9Wr zda#>Dce0(lY1QKq`zPz_FXr$ixh`jtw6dPuy_(pfea0z9YLCBSfG!YU2;ByJ@GV8M z*rKctTLb&sVroqz#r+$FG7%mP1;T%H$sMW9I*(={cP^zI`19H$#0J0oF~+y~g4LqQ zx9jR{>V4q&)608O%M3-=isVaFth1YM(N)9*W!Gfe`PC>7yp|rB+Kgp1)DKo%Q$ABI z$7>?YHtAG9)2LSeN%ju`#)AcFc{o84vk>?2&_adXu0a25Y2kbbmo3>r z2b%Yo-Ij_x(~a_9OGHLf>VfSG$w6AWn_F*K1<=WvrR>6*8i@uq@GLMNe)wq9dZt4U ze7E*{360dIZ$V33KcB{ym*s&pQ~(d6GAj_{(G}_XW3iSs?v24R#Usw?`iBd}J<1I1 z3KEaqIQ9=BqxggJ2FlxLsk*&Lxm$m5fsS%6zTN zuKvd0_Q@|(y0cB&ynA|spl=^1AHL@MW1JS_bmh-$%>;SV5G!Mag%BM&BCO(dJE+Y; ze`DJdEkwa5>cgDCp&|s+P}%$Q)Bj+%nPoA7^4+{c6EdB2y)1^2K?FvTI4bD zpGBk~{@WCT1OOp{gfa5ZmXJ?)!?lOe2p(pVy7Q;JpApj@P!RBmcpGX!rfYLt6kVZ0nFiX@~%r%w?TN<=O^z^N|RL(5c=W+>(3JPSP_Gn505dYX$D|Gal3-wBhe=0w9p{`X5j(9G-b{j zN#c>tD^Aez%Y#IjXKLs{Z{vZfSenGzR_MWRUS}#w`s0*3$wHBnD$j*44_dVFFP?w? zuv2gd@&PX)fLTh)W9Y*9nMwkj^uw_*iFWpc&QbF)iIaB8C(*|a&An3UHWO1K|2os1 zVCfbKpe_iRGyaWxR$4*jWzF=MPWt5p?{)@77sU|jW~M%VJg}%CR~Hsm)G}`V8Gmr` z;B$d9+n1&DqWz}hLgl_`pNV*-W!Ca_V|VymMwRH^THoBByng zl+Hd$6ZvU+aQ{H;%bOd;?QIc;Z3B;WivNk2{sssyhp@iH^7TB02k!~~fs5cNW%Q`F zsa0NoNIiL{#SW$TR%Apff2JEcOZEKEQY}d@ i`M=QU6y<$4rs7*A*-yNAt-N3Nbcz;9Kz3YX-v0oIA>^O{ diff --git a/chapters/10-estimands_cache/html/sim-finite-sample-bias-2_b2c7b4df6c17916197a448bdcdd2e5b0.rdx b/chapters/10-estimands_cache/html/sim-finite-sample-bias-2_b2c7b4df6c17916197a448bdcdd2e5b0.rdx new file mode 100644 index 0000000000000000000000000000000000000000..b2f1a3a1ac6587d3cd31c33d583074ce6799c939 GIT binary patch literal 179 zcmV;k08IZMiwFP!000001658z3c@fDoV2OXf*|@1@4mnnD0q{!-9oTwOCmnO-}oRu zqi&O?#f4>Ncb1vG6aZ34DUfDNk)2P+{Q*dJUYHSU?gYNXYnhpg+k2jcc>=2%kmId@ zL_V5!%8+BS32A;QORe0*w~g-fovu!qp=Hn}Dk*b@&-4j`v-nPC2pS5X=KD{NVfjN@ ht;&|7Z(*q+OI@{Ws@@EioTGX){Q;FwYaAK@006>vQOy7V literal 0 HcmV?d00001 diff --git a/chapters/10-estimands_cache/html/sim-finite-sample-bias_7ad9b5764634ecaa2d807d51ea952c2f.RData b/chapters/10-estimands_cache/html/sim-finite-sample-bias_7ad9b5764634ecaa2d807d51ea952c2f.RData new file mode 100644 index 0000000000000000000000000000000000000000..2706a555640d776a9e764292bdd4a0fcb7e7d5ec GIT binary patch literal 3269 zcmV;$3_9~4iwFP!0000018rA%Je2MFAA82WOoZ|vHNqIfWG6&QvKA#V%QFnKnHkI= zBH7ATl2=|ODodC`iIlQ0=R}dBNR~oHWsBbWJ(kn&yyu+1o@@Pnulu^L`}4UUx4m8l za4$Fv1`~h@3i81Ogdh^6P`ii66a5N&FieiBv5z-}KFn%Ts zhAe<&zkSAt=lof|4hYlZ*`eQjm`a7Y^SVG!fAc$@;CfTD>FPCRq z9n-=S-q}o!^@_3WorMGq^h>Ny!H(*=J>j1DmFZ4O$~??wq}_Pc5FIp9B{R2}=8ZWSYCoeL=km zDE`nAvh$jGuU#%D5bfXc;+&jgEX<@Vhsb}mFwXC7eQM+lBlQg7_}4)=HQm7mrczDnb{z?L19_J+4*@Mab>9Lu@`ICN5x?R&3a}HQl06Dt~`;ZLD}+ zPkHM?ZdsXg-TuEu^SkTgbCCIxvf;`%#>wg`-8-6Xs64RI?edNKsA4{F7_$V&o#^E3 znEi9#fw;uLqHo_rTXPkXf*f*{qA3B!#HWST?&vpDt=bd1J+!QwYF)dOv($#<%^r{) z9aXP>zWkB++2F7d-bF!GX8D|KYhjChe_ghSKRQEz6Wn5iLG2D{tIW?Z@9zt}C4-%6 zzy2sDJiop-dK_z;j*z@0zO#M2o&ITf_x8x~!!;?};`k5g{b_I!7(P8!1^>ErbGS+T zP?A)8q;|e);WdKI`M{ZpruQC}oxx*uNdn?Qqf+G!`%hr@6ltPFo2PR2AL`gH*SRz{ z|M?)Rn&@ObLf>rH*+c4XM=}X_y^mrV9yZ_ke#-mky~WcbE@r^T;e+G!JB{q8NnQur z&WDjK_#E#o?VXUZPFP9_v&V^O?McewPfOIP(ddmEd#?`btHl3q!l`j@>c zBA=g7l6{k|X6v2Ml=ur5g4{O=i)on+C`F@11!NgDv?*C2_9}f~HjrvP9cob>!!+FQYyp!3ZJh4Y^8nua}4w1fBQBIhtIdL%02ysx2$icK)%` z)Jc!e!Z(*GeY8 z$UN>_6TH@JIzyxQ+Lf=^g}gndgOT)!d3zvDCw#u1GlqFuAmhc6%l{!c7w7z-d|~sE zSBPY>vc9LX@d-P0at`;Y;z}q@-PQz#t+3R>+|R_vi#NSva>9Jl3V`7Vlxk%6E>4Zg zwHQ^kGsi}3aUMbStnH5Wb)3WwIzwcN%m4HKoLkvk#M}_xy)P2=?&^RA>T2lDSpw$p zO+@vD+nYQNii{lYu(BV{7rZg56kBBef)q`hMqI)zw`hxgET~VkQ%oFHg=t@rl2g$> zuHl?2dM0+5_2|}@3L74_u0XF~V0w%282|HIS9qq<#fM5p2h;)^(?4&`al@ZTH57d% zL2PZw!FAsqS3kFtYwtZ9)HW$UdSF-uQQhD&v(y%NT2R*4&_O(syG1%r)qPsOxjRHk z{;O334tJAeanc=kb5Eb^$})Zaj{IQ9Ga+qCyg_@$jt*1QH?LOS-G2;@lvJBBhP9)6 zJlykgCbpI>Ff9D%TEiHfg`)`NGkH&4PnEEgvoyjS&Xwr#o2wMJxJH({Ti*{>yC9P< zSv4Kjo1%J5+t0I9F49pryZXK1B6^Bkx$P1I6|y|>$xazr^3T(y)hf($`OGr8D)(#| z#e_R`oXu?GwkpN6SswP1!v*0nJ{Dr~7N7hqS9o%O7KeD>WYD=+iH3>0gUOC67?$-O zFt>MGAjysRHg7|@#FCp0k5?9WdDn@JhdZFseck8v-J@|X`XLF51-9)C)k4-^3+dP}<}e%mem+5xZ|Jh@=E!51tprKMy{vgGt!lFIfKKV#=kpaXhtH0Zjb=Zz z#VpUAO{bP|YuXlH%y;IM%*l%U8hRT3xL%?F|MWpb+X8_sEay^eXfA5+^k^EwY1t$xG7>b7GtJmHbB`;qSMcf7;-TJT|;&BjEH% z)kVT+`IPZ5feeH8uf>izA6~wlF@Hm7uih0$kW)Xki8fEu`6$kLtgcwsA+Hd{=y+bt zr*Pbf*{1MEgq4hAV}j`ogwR!&c}vP|0jo?4UHWA-if6lHOveq~kbcwAtWAI*1J&Wd(xjuM)=FeTgitZyzWTlBPEm70%K*B|>j=N|K7ZoSFQrNb$Zy<@bd5w2Ozt0xQEX^Qlp| z65dnyZ1Uy1#vZ<5Qp*0GwEEzP;A28H)8Gm0*JC=PRomPv_+Jz+t=#_yUlE^P7XL^Q zpmWMiimn=Q{xQ$(s)O^qFfW5rzo-*xv-rOEKi+ySD1Edp$YPVJ(36klOTZBVK)nDq zm7>QYQ*}vX8kr61vT#%e1=PipaV%dW9MJ@iHQH^s+rSVc;BXkCG1`cT)dz7VSPTI} zf-t53A79|2si`Rtfg*qu%1Jof&(ALc#RS$=T>_m5o`gd*fVnO}3ZW6$WI7E=(*h#k z0HC1(5SSp24FWg-3JIY=A`u|s*fumG4eI+F^a z>8x2ekc9xH`i2F7wDlkWK%=mLHWaM=I~cG8&_EQRud@;1;da;#($Q<`1{4>9`u7~j zsP&%(`G&Dst3~-!sK^ceC?I@2fAtIy#NXoAJtQ2JOj(7;lWBA+8Am~?GnjNdnXSG? z0z_J_h2cW{sUVGwq=8}o6ANH-86b+m1c_t<8}h0%K^B8fV}a_wA*dm2sGrEyDlLF8 zw(qKfB9Ylpk97c!)+*iVu-69jU+3DuV1-brI3_t9{8u|c5(!)*ggi(tunRi$E(n~Z z1?a7reyh403Bd12q&}bv@%}yj@5c2J{EM6sLZKj6ZT~)Y8pKUoZ{=F`7Hgli`9Qk| zZIm4c>M#uiumF(^70ZOW2yAQ+3pxl(=l7}sAaoG`eKZ;kF;GDk9NN0I-yx*F7X0rm zfl}5QMa40EiKNv^m>^@lAOaGL)>%c-0kG(GX-AL0s5q}-Gj|Gu;s42iGOkTn?F>%c1+p&odC6f?^gl7)B}oB zCA(sxSg-$J@*!gxHDHA+t8m2C!89?F+PU^ZFB)>D%2=l74T8QbVaec}dl?LG27%XCQd*%* zqw&pU5E_a!&&trhTjEYJd^hsXM`QmwxC%8!+_66y-3%wbf!>a)DPq|%Rq-yoh*vRI zwewyOhW@oCf7NPO)V?fPnI)Tp&Sl`K)2?E0*csjS`wF&dz~&~mgMl{=-YfC#JgU3j z>-n=33%Z(t&N@@<)=j{#6=1^vw7G){v}xk}LIJkoz!N38CEfZ^i^QMyvfWqMX~*q! z)s1U+xgYAf)Y{Y8-HY=ct9Cmp{MujQml!`-GOp%Ne_N233gKZKc%(lv7e~~(E6S8S z>ci2Bq#QGI(B%E8;r)|iN;(l%wKS$~+{ZM0lqaTzeh2-I&aMT$kJ10LRVcQ9af+(n z-NN<@=!$m^^r8(VI=Q931N{cU+fi#xi5|z=GyI1I4idW{q_6tkL~NDx7^wL7{EV1P z0)7lAM?-7(FbwRr15k@|nCginny2>E)h9t>L3QblTK_dw{U9w&hdIAE_V`#F+_e+pYZ@8#uOT;l2FoUR8KytsQ*rW^39 zmwz39A?jC?uD7D5ew*es!MvI0yP&bV>Ad0i8jV-o%uktlGcQH`hP0*swtvPQwmv2N zmDT)PwZBfq@@%UWYw9viFFG94`ak4mlT#1BBqMA(}Wy=~<m#l ze~$Fujr#ktUty0=$yK{6t)o08t5N?TVIAlGRJWK+* zQQk*IU!2q@Tt0{X1M+sF@HmQIj_3)zxbx7HGaqwP4b-~2En(Z@RtdH0lY&0 zC&FvuqdZl@Q-Pl~U}gRrs80UBws z9ilxzyN~t=?J?RtwwOC_J}i7!MScZ2kThHT@+l+>p*pTcq`uV4K`15|8Su{so}Lho#sUr+5GW delta 6334 zcmW+&cRbbq_ty{__K1uwGAg^WuQEb*lI<3ukZWXL_ua794WDeu$jG=hg?m$%%DOgL zmrJf0H(d9+?!CYJ{r!8+>+yP>^E}Vz^So+FC*KvGs^@J5PCADdxZ0Rmeplvy0gpW+ z0BTs~Ojp0|a9LXL!E4sb_dfcI5m;-qEIrW~L2R6un+}buXGA|NHiwRT5m$0YsA=Bv zAZf`!(YJC9Z>7fwfuVOy!e@97IsFQz*-gyjNV*Q6K7YD2k;#_Leyy`nZoYDMw_WVT z9DZ{LTZBM10X(3$UnzkK272qn$3fc5YqEZFPR)M@QKh!0W`>hY-TJS3JDrmJ z({iRyBT4;($i3kY8rDNualg#vWF7v~wW0S3K8@NxykBz?J@qwlBo)soo~5s!zf7*= zzb5y$I9>m@R$IHWNB-(GckJ`T8Yc~q-`TJ+WRuRCa3JvH29%8$N&MRtI2wxEMG{O1 zgh&QoDADaPo~tkRYE0yMXq(`#nWq0hKIuvg9SVY8L4Sii2MP6>s}lHG_b?Wg(@o(8 z&uyKdW6JzPJqhdrMnik2eBjS(r$S5{VxmhbgY2TZuIGAS3CxTP0P4| zg1@J}_ON2El;)jaZLU$3<};}*PkQD}61@B;!&gxLYEnvWK%SDIu+$J3XWX!HqLkw{^z1Fgds({fUb}RQAy4}jG zWHKs&8=Pr5kzVzq>5qHmr!)pwCgG{YTnUJoJc(N+iBGlbu7aNaY8y=XJ}>H!r+V(p zg`a2QrfWc655Ta7k~lF z{3ztLcld3O9tVAykdniAiythZ{I3OVGBQG~?mZlz^L_)C8BHC#)=Swe{GjP9b2J9A25Avos@2^5>yCE4^+ky{~dqC^r)C2_It@-`A3?*@WZAg6R z%CGFmdbjH)uw1sVq}Y{4XuE_##HE^1^2BS_KDM84(N$LIbcx~4lU1bn4cqI2ZEH3_ z5eFt)-qMaHT5D8wbdAGCV$2I|gdl(H^WG$%usGJ5c*(^pf_fOBN(A*t_Cb>zVvmwJ z(jlKG$EuCt0vM~yhS3&~@+x!;_p?+_?cZ-dp+w8QH4Ic${XP_nIJLk}E_xS{5ZE2Q*+D0^6b{n*jyV>O7V3dmKcD#8c3q4Cd=cOvH&1_!1Lhfa>x!>nlARex{?#-J= z-{zrqI#gFgPa-da%@auWNi7cHYtAMBFK80CZn=8w6C{CQJTF}`HVpjofIsEgcIa(ea#fx_?9ZeLOB^(vj!hV|K4C;aC zT=HcQ35n#!#vKst8UW4%0d991U{Uj<#F#FI5t9E$4@Wsv9GP)6prqVYlEsPIiA9D9 z#~qlG4J9cI`2}^2N6+UOI)$7=f|6OX^fBn+`;f^)jdg6;TKAO(|8vLZPx4K)yAN(9Oo131^K(ZV^PNnIE}V8dumWL@ zfp)houVit`_7#+iO%bF{vw!|_Xy#Ehdjs8npE*B^EK+p>+L0O`#&Q{t+bF|AcZVg9OM@t|2yh14A$eYidP8&7`5XXN(sE2`|KVcHYWcG9%A`$w~WwmVehZW|pJtIgz z1!on_>;i5Uoz2I%oQgZ!^X>*pPl);^Gf%>e2L2smU1gX1f%;%RdONL+$$ z9bFl;I^_Y5!~g|kAAJmPe@p%VD~UNK*CND;MUDFi!r|Bh^zWDiMvCXLSLQ;14$#@T zM5b*2;aa1x;4YqbXQu#$^~3>>Gp*Z<6}ail6ApdB7{52l%~et0OTJK zOCD7-#||jT>zQtPyhU4eUMD3nrLDrCOJ*xD=Z-Gv)Qn;kAi~r(m+kP$91URlU09+d ziYu1{;fpe$Wir9qbZ0Poz+J*L6{7$l?Mjq@(C+aQ5UT{DrgZiRrd+uAa33`E z1aL8dP2)I|%7ZZ1KE#0t?y1_0H~7&yAVs|M3CdTjE&2oI5t~G$ zy@0vw(b`OCu`ycUIv1|!c+v(BI)N;yKW;@jE9Cu&?Vh|khB6TNi&8)Vs$ z`4z+HKaO}HPWQl*_5xoaKlFkAtHt>vJvU|zXD$LvO2OZKj9&eoMKPjgF3ovvaPe)> zmnXR)wUvbAVv1`u^(}FV+nxS1??tx5{!@UH*b+0GFH%rL*U}%*Q#*5kX-Kib`QuDv zDC!9E8T}N*XYW@>1Ri&oi2VP`VdX~;eNzY0QCavmLPM@K^t<{4Xd_H%VWvF87Gr70 zkkntnnICxT&gi|vPdSl$vWTvR$GnAlUW0$QOK2a5b}4ZL+0E<-5=03-XQ~H)z~M`A zLt(uV4={&3YU-9kYB+~L@cxGLJ0*KSbA9kw$H1@U$RH*M4dHhFA3uEXdmMmGk(xqe zBtzB3iSFjs-n`e6t?o6nBhY6be$-}D<4OZD9o@n&nMu{{KC6l2SCRA(tPI$sk^nh0 z!^E?zps_sKRy0b?5qtNava@H2I)(ss2h+{avjHiBWw-NtV8LIeVAw>!`^nI+5;`U*LlEjp+PSbhC33@+# z(O(-zeoKiw#i2j<;jdl<{048xJ^daWMIfv_m=3Cf<{$61VLDMGzjM)QV*)ybHr7 zpn@JC<1UAm#>Ug{j>fY$0CR@0SMEGLXvZzVa|m+VTT~7=5qbq>n2w!lOPu##;$Z-x zqgIlJ$2&X-?-#5+XxlAC=E1OZ_*D=dXK{R^CzqdL?0)DJjzMSd+r=aR5=2?~N$X>3 zk9hrV0)u7Um${|@uUtq=8P0gGEsdK()2BFoQvZoX#3y29F?-8A_;UipV`W;@13z@E zF`Q}#9U4EOzrUk`Jvz-r;{(3XQ}yz>X;JqnA`=rD&(KAm(R@)^K4%T7Sn}Ekz3DxT zBXSoTK{8eXZX*dAzjRMgxc+#fBv6%f7o!5D0Xp^d@{>PK`w0w5<`3mw5|o*$p|2<9 zFjlwd-VJR`FabwOWUWJ1=1XX4vFBK;@wYw@#xZW>3t3!X^HUv zW#QMQJfp6xlk@FU+xA?`wJGEX&k@#AmmqnVf*>k8c0WLh%43MzMO$}o~dzv(oJ90vB##U zdo#Cc4U=oVM*t7@-1n*n8BqM@?fK3x9@&;p=gV8f6gH8w zx=Pl@jMVPD-3BmDOyXBMfRcvw?lh$>%DL(H3d3EfSPf2v?$!&n3f1;C4Rhf-3pr{2 zd$PUp$x@)QyRyd1;L++9V5K|$)qksi9j2Lqca2hX=tjUtan2dn=gpxY@|J{h~B zYg}{OyMHdSc8#E7AeXvPYpo53HOhfDKy!nh_;y;zl}^yV>gADF^xEiCBRtw~`_vtq z7P@1$?xN0V++@wz{^M`<0+u8_-L9GSmNtk~s`{m}`YaEa7fvQ)%^9WzBh=xqkDyR# z@!6%>{JFK&!I6PHHp4Y-YNlF19dxQIrFKEeuG=+0`WN|@=BIMihm7f?BAA1hU(=r2 zKQWbxz>u{BpD^=LF}pme`L~)Mo2YD2u&^&Y%c(CqO~CG1Ug4_Z|23OrwpD0GPci=T zG`d`6`!oq`H~u~}`aZbyH|P3D`&QS|l7NzNjo8Smh+mlxHAL2Dc-^MQ$N4QWk)=<; zRzq&7r$)x_z8m?SVsj|tl7|{Ic|-5G?+-0Qmqs-~JYI!2_0ux?7X{B>~6 zd`~wj_|AHZiY(Hl-c?q%{IWfg@SfjO^r!4w0648yYX#8bdhM=uJh-@2@gV4d&+UNB zBumeH%InCJ8>W>NABJ8!KgcP)c~!W+_Jqus!fnUX`MlFB-Q_8MpFTLBDzqD}k`fzg z;0VvZ>}-vE^Ze9{9UIExX@mUXy`(WENaLr*d{jipwvdGSX{Z=MWmehIBc zHNFJ;@m*|$sDQ2){iwc|5QFIcuLm;<)K91Aaop%H!P~cBvEKubUP^9-A%l=7)hyb1 zYc39~{`bZO{E%HzLn%)#NEEApY*M0ump1~_(`@;X`9 z#%1+i*ysG))8o*yc6QvVe`&{44?DrT?E=97CL!+u>y7$)VYs-jdyy*U()ioZ9OoYiPvyc@_1``*)Nt!V*mG<5b7aHDme;U0&!1 z6rs0rqn)j>SI3u?I>JR%CPX;~-0TD&iC*NaEo$9v9eYr~{yfpL!#gD6?B3a+DvxSl zvT4Q@REGNaTvbhDx z2Qvnnwd?jzUHUtIujkliyE*R+#w{k2TZ()NZ7l-VYM1IkDxm2&*psdm3k@U3mxLtd!|of1-@FK z@luqm_2)KCLvnVz%JH@GcYbrO-S`5pY=HXjhSkW#&aUeMEtcn(y zPAv>>-N)|TA^1!N1Nc+d_r`)63Q=hskgo;Wu|~v+S5N_>TEXZdF-75}#9;;cFuXW^ z(Rb2lFT=m7G~Fm{tmvp>_2WrdU^S6nmQerJy^KX_F#(gh+&j*jSA9m^QU@(8DS*X^YCy2SYd4adUDAasZqEGjLQBEiz`$ND)P=P6b>SI6Ljk;B8@FW zLU%SUCjFD%N|;ov2lFjsh&+)5ZH z;Qg6Y{&*(yl$$R(KQ6--Cp488@sBG<&%e{`++v&gUi^;@Rt&Hq7VsFSe`%O;71MZ@ zs4YEIs9Y^uJmmg^K15>K61Ehndn{^D-+x)V91Cvo>V8k8SbjUT8j~D$yCCH4lECR> z$!`RjhbBXK>C>*VrSQ^!FTGh`L~#C=ol{v$>xM6Qn7u*9tSKaS*w^qt2Zas({`eaI zE=2B{^{vaCc>6oRWGja!U*83}whwc0L1I0NJyA*zOLo=m&k*U;nJ>w*Ah>+Di~^58 zc3btbgY3DCOm;##+fuj=Co6XvcE?KPuBqfzoiy7s-M0j|Ul=+`gwH_(fXFG&EL|FC zVUL_YE}Z5G^O(i^-aXMLhKJ0q?oLhRqAp&4jZ@v;)DgS^a-TfnRnzMo+q{@Aoh&VJ p|EkGDfyT5ll@7zCYluX0%eG7-@105gpS@eW{wc@j$~*#L{|DX51v>x$ diff --git a/chapters/10-estimands_cache/html/sim-finite-sample-bias_7ad9b5764634ecaa2d807d51ea952c2f.rdx b/chapters/10-estimands_cache/html/sim-finite-sample-bias_7ad9b5764634ecaa2d807d51ea952c2f.rdx new file mode 100644 index 0000000000000000000000000000000000000000..d68a1dc206b8bbd97927d0329c12c8afa289f389 GIT binary patch literal 178 zcmV;j08RfNiwFP!00000167Ve3c@fDMSt3K(SlHP?GfC&bma*Y+(}xe5NO(xh&R)7 zcp`O@v{nPd|1%%n^ALP~)&V~gx~*l)H#vI@ahvEWG%-}7f@f8Wjfqh!V!kUkTT z$UU@Eh8&wRwuPgtv~r_gHoDVSIy+^CmPMN>N!hc!#-AXP#e0MyW+*~in7@1~%WukR gRkbv63rh`I>gxGY_hzu<99=y01N}k;)*1l-0B`|OLI3~& literal 0 HcmV?d00001 diff --git a/chapters/11-estimands_files/figure-html/fig-finite-sample-bias-1.png b/chapters/10-estimands_files/figure-html/fig-finite-sample-bias-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-finite-sample-bias-1.png rename to chapters/10-estimands_files/figure-html/fig-finite-sample-bias-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-finite-sample-bias-2-1.png b/chapters/10-estimands_files/figure-html/fig-finite-sample-bias-2-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-finite-sample-bias-2-1.png rename to chapters/10-estimands_files/figure-html/fig-finite-sample-bias-2-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-ate-hist-1.png b/chapters/10-estimands_files/figure-html/fig-sd-ate-hist-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-ate-hist-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-ate-hist-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-atm-hist-1.png b/chapters/10-estimands_files/figure-html/fig-sd-atm-hist-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-atm-hist-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-atm-hist-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-ato-hist-1.png b/chapters/10-estimands_files/figure-html/fig-sd-ato-hist-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-ato-hist-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-ato-hist-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-att-hist-1.png b/chapters/10-estimands_files/figure-html/fig-sd-att-hist-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-att-hist-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-att-hist-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-atu-hist-1.png b/chapters/10-estimands_files/figure-html/fig-sd-atu-hist-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-atu-hist-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-atu-hist-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-ate-1.png b/chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-ate-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-ate-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-ate-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-atm-1.png b/chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-atm-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-atm-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-atm-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-ato-1.png b/chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-ato-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-ato-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-ato-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-att-1.png b/chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-att-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-att-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-att-1.png diff --git a/chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-atu-1.png b/chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-atu-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/fig-sd-mirror-hist-atu-1.png rename to chapters/10-estimands_files/figure-html/fig-sd-mirror-hist-atu-1.png diff --git a/chapters/11-estimands_files/figure-html/unnamed-chunk-33-1.png b/chapters/10-estimands_files/figure-html/unnamed-chunk-33-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/unnamed-chunk-33-1.png rename to chapters/10-estimands_files/figure-html/unnamed-chunk-33-1.png diff --git a/chapters/11-estimands_files/figure-html/unnamed-chunk-42-1.png b/chapters/10-estimands_files/figure-html/unnamed-chunk-42-1.png similarity index 100% rename from chapters/11-estimands_files/figure-html/unnamed-chunk-42-1.png rename to chapters/10-estimands_files/figure-html/unnamed-chunk-42-1.png diff --git a/chapters/11-estimands_cache/html/sim-finite-sample-bias-2_7d0b601f17d72438f53b6385725276eb.RData b/chapters/11-estimands_cache/html/sim-finite-sample-bias-2_7d0b601f17d72438f53b6385725276eb.RData deleted file mode 100644 index 3b23d25d7de05e49843bccc9b7c4428ce94169d3..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 3337 zcmV+k4fgUMiwFP!000001C3XCIFxG}AA82WOoZ~LNT2AM5&iStE`{#Y{=iYwz^Ly_5x!%`h zmxnIY0}6pa_#piJJP< zybK5go)5}CyNzH^cr(52V8%zXLOy#j6bf?YkU)1|(`&B$JVW^3B|7gqx=|d9F zmS$S)Q^Vq4TTP7ih_GxO1^9J!iY<>r4r#kQ+`BECE}`~YPw+DivX+&a-iNJ zI=gA(2LOoaZn{4I>j0mUz&TDER|9X0tGf_-Gb@nD+Y?n2 zBbwV?)-s<{TIyK4=l7AkuDZBvc%Ha)nBvtjvZ_+o)+Q?om)PKP;c8uE5f5<)vk1i= z?O<=6`E~c+*aZK=&tF1Xa^w;N>~a*M0{x5#j|!?>(Jv=kw8oL$)XZxto!b>NRR(2E z?vm~8l`nm`@Q(07_kbbJNlsa6>9ll9L9=aNZI+NPI-QRl)NF`B83eahUCVpOYTiaM0?fsB}dc@d)>f|l4y!*9()jbCcotUhIe%ib- zY)9N+qC{JSR-SUfWxUlH|LO6@H*V$~L8G;ae4+s(5@q#!j$(EeYM_LhCbReKZ&#P; zSR9@Eu#Z_qaIhSvZM5m=CUv#J8TebChcNZ`nr?nM?s@3;!iixg6X4y@zA@U(2G*lQ zkA1CYLP=&k_O};zjZ0a^FD8fDVnudb6gqlxLMSO`sN|MYXVqz2E2~#;sMXEl9*OV! zmORTNo*Y$>ewn6X?HS*g@Escr-#r0|Xe6{@d~ZcsZt`&M_vuwo=zHoxQ*u84WnTdM zT5ecWQTWg~vioA}=L3oAu|dL(1EYPnPu`8zFaAy;aV`%KjVtM7u~v19Ej1?J-<-9H zuvK%h(i+a*D`P#`g;%_O@Z0u}BVIy5FagR{nF>{pXRu`$jr-Cuid>GMS-(rsl9p;c z^T1*9nA?Xk2H%x>(=#Dr-}e&DF6-F^}@4JlHaMU&Uu*9q*RS zZ#?)MmLyWz`$#%2ek&sTK(8{kIIywH5>K}ll$f9WmhfQVnrC!&s8?z}Fcgkbj_BIX zu2#Ant*mnL@US)3EufC6Zf{%5PH3mmg(f+?-|o!1l+K3F4&q#UBT%m|^_!tCg>0L_ zV-8${Rh_-D!EK+=@PT#<+o3%Et0M|Ag{DtQQG_YjdF)cNmhij$x&#~fgb`(k)

! zC9NZBjw!+?V}_XbuYWAJ;_BDtYv=b*Z4w;ieRBOG*I2S>fAL7aiho1eht1h8xT7h0 z!q3GBEzQ~3u3KZOr?+uzJ!b-1CuB$V4k^K^>Yb(+Tm4V)OMC0tiAHcXN#-iMPU$pt z1xv_&vZ%*muaV4-xni&F>~&sVqRriu9cX_npcNRW+m^ny-5B-RqlJ6xw}Ii}DiivU zR#dl}Yi{=V=F)k(neS{%D7~X#1g3a0_mT7QVy0rITBzOWVr^birJ`o%h%#5pJ3%UE zrSilpr$T#@l@DwAxR=O8*b8P=z0q4hPm(LPoTsCLm&V`QD8h^XbE2e5iE%oQQ7TjE znk6M4f3udok!93cDW5vS)jw}{R&bPu37fdiBRj(poam>6D6ErCaJ+oJbz6OvfaRwG zo5by;cS*=j8I@Z`u!-<`#F)}dd%dRi zye#o3&Ug0Tf!`FQ7%5h#c8{pXi)vE79c5u$^}cgj+Ee0_63TfTD*7@xsNwp9w+}T$ zr$6V2?@~8jc6-r~Ua93N>V*jIj!`g&-1$vulJ3+Yd-X*m_Z-swp~ zozj(xLowqz)Y07gJ{*VV88;T$DJiKD?X>eA8ZMZlTeL3b-470Xv+2yEdPn%-p#>{G zhx^J-Vunk{jlT1x>$ZI=vd@0|?A5gCOMF|^_E@}(>hTTKIV$3vDEon`d~Lg|TqM2y zNfD3S5eG)A+%Mr4QuYn;##dnimz?Iz18?wIWSAjo7tkoK_0~~@3%Wk-n!QVhZ7$mh zSGDXMFJ)FFe(Yg>n551mE4HjoFBZ)99)gX+#uHo(Z zXze|0Irpr3A}Y!mF2)eddYDMhzh|877jK$mD0!w^bLc#^LQlKw;^uFcfePiR-Fcu92rvb$sU0XJIqmAQ{y zkx4sF6P5BM7j@z_L?@|HlJ5xps(Dh!xYf>?Rx{Y(SChRC0t&&qGd9lid`X*;-jYg8 z?((zF^-bXjUPDmD-4c%!-$c}Ew1IfPm30p@BGG7$fKRf zJ6s4Lj=M1K}th}B9=u2umI>190-a8fPiITnM4+7U#rb~t+tf}EHVxkNQ8qX00;xj z00x!Dpn%yBs~R?_!GodRp?*Z_niBw^QTl)u=&bd_888RXKqR1pSod&qIbZ{>(W}e# zR2(qXUt>U}=HCnQ4rMV{it-Jlz}N93fv~msl|6tde)3fC%$dH*Bykg-B(=sl?EK%>}SHbRvq*AQH%U7HCyv5Serul}S|nk%AJ; z0{Q%d2e4T06{gKVX+?meNMsgBHv(X5B7X9r{ozCVtIrxc;B%!#RuA(ZT&&|VgDDg& zgBI%3?hAPH+Z-{8nKcE0qCRGq&0C}Mz5E?3TC#d zg02r1m=&rCh5maBtwL&;Yk+>f3|cVV8^?iD5$gjN&_(zob0CRE(oIE;lZ2qCX`-!wct%&>#GDeaZO60 zUOWi>8}aNH_znJy`gXI~MQk9<%*%V3Np4C2IkZ;DM?x{0Ur*03f$Xe3=n==VGqg*+ z4O+N*`0BNK{QwT%0p0W}XcWz*jw}ZuC(*4f>V=`u@za%YX3e*OY+VT$n@)1!0^eN{ z)nPU`+MSfI{@2~7z{{YYrs5M~GWFAi$x_0~I8L5LO+N61))bxhP8>1av!C%c>587W zBEz1U^bZH@HskR{>_8dc)9gRp>4S<5c|(h;&sBO}v1hj&D$7S(dk%r`noEBH>PT{U H&H(@bD|vr$ diff --git a/chapters/11-estimands_cache/html/sim-finite-sample-bias_cc81dca70c2ff5532b8aeb5e3b3a3da1.RData b/chapters/11-estimands_cache/html/sim-finite-sample-bias_cc81dca70c2ff5532b8aeb5e3b3a3da1.RData deleted file mode 100644 index b468e1f7807ab615ecdbd500d3938b5c65cd875e..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 3270 zcmV;%3_0^3iwFP!0000018rA%Je2MFAA82WOoZ|vH8RE+l&m3AlC>y_S)O5-%`D6y zBH7ATl2=|ODodC`iIlQ0=R}dBNR~oHWsBbWJ(kn&yyu+1p6j~q@BRH=_jO(O=W{=9 zd%X{%>qFpTJm%|Oj_&2{er{EX^+g+MX6uepx>-V-UCE~^|&2-VY*FiW9y@7hC z=Ay_3^gk(X}X*Bf$6 z&Umuz`lINuytQ|u1?+(D|kr%Ps5AA(CNua_}9%F z!*<6FCd#x$=;o;xTqD?=51bxveD7h|5j8v_uZ)5ZJJ*~P~+AHx> z-;#HE#Pbs>@^8{KY`xKy=B;K_F(6o|4mTKKblZ<9y7F?Z z=N~&wp7i)!#uU0R>Suqno^onD$%KmcJ~9W}aoV(e)BfP^!c+GxZa1-%6L#ix zE@ude&EdW^!fVW?(zS}NUHNJ}pS$ODFp@q#XAh*J!shC@qu8hUa$a18ydToDvCa?5 z<~JUBg-DVp?R_dA7rzaaeYjU0S4?T_vL-NWMP=q^eMXPI^IW}yI^9ZVAZFRJ-!JqeM-AZS}X9w}_y%FelSNko{S3|bX z5U_`DBC0Oj-r#XiZ1`}ymHkkj@Qo3bm_qXxq$uJP;u3DDSy%jHeqDl{a>9r@O!tb6 zf|~AeE$0;RGciM~N4LI|+whHR^Y!!lr#6X>3O>Jeg>Ncbbf|cwUn8&~?eperH~fhd zBk@;K#Fpl4T-V((&2!s%_TDo=trLnP2Zq!TRrM~@i>-mDh2?#X93&%nn`Cp<-KPwi zx;Qe5U#;qKxSJ%4lkT{idwN}$m*{hM6bIU$iRe<|4BOJTwVR^9dA0EG{$pUcxXO$% zq#M=k;hvj4zPWUsVc|d963XZ(7(u9>$$jd2s+gsksTJyQu2^5tT&<|tHKNSj`hKv+ z1-U%w%Bj$vWc6dZex4-?5sso+Rqu@!Fq7npEteQ*&eHfNJ5^-yKTnrbsWH#xF-sLH z-LvGB%mJO|>po}T9))u;;KVEE+qTtLiCBLv zuuI%Q`k18Gsi1M!1Thg_j~Y{(iQM*KuRzDe=zwP>dTh@hek!AifaouWq2S@%)?*L< zLRI(#HW&4n9y)s9iDq-=b@-uv#omkYCECFyuCn>!#%B?mk3PFx<&b@#q|tM-_KpQC zvm=zCU6|u|Qrn48|Kfn&Pvhl^6Rp_%!Tsom4*l>4T6Uoo<6EPbq8FGd@?U2|n*= zJ#R|9it}9qcM~=Rt42!IX*~pu1kp|E_oA#!t3Gv3%X`awRYSXu!zJG&2RGb${NahV zAhP`mpxuLWI0FdR~k@m<<+)4up~sT1E5iCyYg*0*arbzW z&}R+h;?S(Pj;$E}BR`(gtBl)=+?15mh<5rVFKsvMi7mQU@*Z)*-fuepwB8wcY-qto z$mx-~i`1^AQzpNJ(hb|b7CB~rc=>kP{0*V4YDX+VLG#oG+8hn_QIh*uQ@OTXQ7Mwq z{=7&)>9`ZKRq2m#D>=u8c+(pQk*h9qmXzBYPY9%Ct0tr?OqQdyr zX9`6NZbVhnrYFX6ZKs@syfLh31)i6{5r6c<;y1Co^N$FWCd5QZboy+#ZhjTpH-OF) z#*X36?n5#rcH_nFAG&)+WO%x7vaEIRNvBKr@kQT&W~*}6y})d|Qo0CI>ENF~UvI(P zUMlz_-d6V9G&))KXr9YbW|Uh~q{#GzN%@v%y|bBF;-~d1HGE_`|Jc_t`m-n+-f^M=7W&)($Unx^emtD)LF$e1zgeI zNKVQ==0uNZM$)rF<@*j5-{{*rH3V)RC>K3@nQAw zY}14K z>Ibl?6nz$%sz)Nz$ZSxLg`+YkpdOx#gAfE010n&p3q!z?NXEv7SiA`i#2bT#cte~a z5oZ8#%KksU&_z>IQy?5o04bD{aJZkJUpSfxth#yxIuSexhhzYAJ%Gfa5!hrp4N21h z!r=g*r3Da}AdU?JH~@lhD3D16h&VQm1+t;_wak3iGFu^Flks>8h=i5^APleom^3<* z3bCP9BU~s#fVTRE27t6R5CCA%#(*va*8L3zECCD<2^gT(5gu-b?I0hs>aJ7qAgX`g z0|j;dGa=tlHftp*e+m`3t{(}6t?geq14QvR{Mrf$MP0aCKJXR+_!qnEyK0ItPnGrQ(?6Fz{dH1W6=tRS;T1@_-%Cp?5&! zEFD09HT2unTi*cuMj{OWJxKTO_`e6&TJSG%28TjHuEhSm?KDW6wpPm3>@8M5tNnpy z51J@DE>vL}2p9uIGL$S6sv@vHK`iJXEY$B&13>5^00tNg22!AcEI2fEtG`1?10DF^ zO9E|KEff{U@FkK~GGT&@wS)*rV+?AA1qB#m)~q#i-Nvks)T)@pY6!kQXAmn~2M+(= zU09XWw$z4j{!BWD;fv?34DVWGy)P^fxn2Y+h@-9AP+y@72#0PYbU}aqA8WCGixCb0 E047Cki2wiq diff --git a/chapters/11-estimands_cache/html/sim-finite-sample-bias_cc81dca70c2ff5532b8aeb5e3b3a3da1.rdx b/chapters/11-estimands_cache/html/sim-finite-sample-bias_cc81dca70c2ff5532b8aeb5e3b3a3da1.rdx deleted file mode 100644 index e2b8c84b5b51e13113f7b706f117eeb19cc0e757..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 257 zcmV+c0sj6UiwFP!0000018q>jN&_(vec4TmyP$~u4iDa>VCg}6@#4Xvw=%6$3GAjz zQVQ)q_%R+-`~#2uB6YUebQK#I-g`62%*#CF08(hJkamQkGrk+0Ujj*QSJ)$tB@%?Q z{iTJ!4qN>>0J`^pu4@V!S>4o9$zI55n6^vmsUhF+v$?Tm$rDdyRS1|eR#M>tUyT#h zVA4)}lcs9;KmUF!aMkLMXW--|I8)z^nIt+ajO92ptH^o2rX_{R2VWd9J+Zstb<_oI z)*}6ZH*pVLwyS>sBHTb8{!_=F+i^kJikzWovGm%SoG=m{}^D>+0#t=|9o8G}5~*cNqo~_Vc@| zWq9T;xn$(09CoIbXB|mNfkbG5;9Qg2U!V6)qxEx42RMREaxcBApC61iI?p%0`_AZs zB>5$h&nY88Bm0`4SCs>#85lb zjVjpKH<-&*lEd;I^yo~hXK0d{W(!e=xwcPJ7E|8LO;asCky)HZsP>neDj2NSCc( z#OB@Tr)nrzx8l87I7#sc+hvi{z(};aO(gM=&n1qD#?=7s;Mzym-loy3&?Ak zB|NpG2&}(w+w87y+p0#t!5s?}p~R>5W#&@JLtg&krkcm{jzlNl+AUyg;EO_kPy<$M zc`WsWQV@O6!ad3AQ{jH~@Xya3=1+0oC%f{Bf!N4d^L|3|3F>LX_~}H$_|49d`Qge7 zTWDCpoyX`ebJJhBFDLCYF1Nj}Tr#}NSVG|HV3=mphWE&vv4iIt*R{ZBn`h1iedne+ zo4^prQm_2b_u*I3>qG)|q35g~u73L3;ef?>J67mr$;)ZGOSNc&hv6f+tu@zUg-Ci@ zds>YfLb{)5i@H2feOWB45?kD5HDk?WK>e80`b%F&v}NmlrOs4$w$7B01_AoG>k+rP zew6}&IsPV<6D~z&n$>cv5;q@8CUV|2(alMCt1h6jjZNPObNOUnjB9XAbNBm3y;Pua z{YD`DdAISQ?~vk>R9Sn&|AgIJ??N=;x}H*O89VM(kSfKK2+w5VQOS(H{s@43H^qj6ysXMZv z$5@<`#+Uex+q)k*HzfM95VclF6Y--DoOPV}CCQ~K&7V;X zzbv~okL<9V`PfJ_=4%w4n-?+bQ&s`}MvS*jllV+(ga ztNN^}NZp!7Kew)F|Acu6*5h2Ny}z^P=(2jGV;EK#w|``x*&e?}ud5LV2w{D-YmXEZ zU~J$T7l!3!1j7MWSm2Ku{J~(@aoE2;ISP+E^6UCY1?J0q(yCi97#w!(%B4H@So4GU zx%X^bxR&=FzDJW>J_%>VI>%%AA-&rCtp44YP`T8U+&hk(*tg$PvK?)FTbTYF-@8DK zJT)!z#BKExHz@C1KUfT7xI~f1a1ulHy$f5rp+)u6ohiUYD zn>`IRd2FQ}$a#C5B5|k4&lQ#ZW2rxf2TPLtU$`X1VkOE%u1?UjkNjfM^xN#I7$?sg zT&20Xss&F5sh<=tFPP>qns8$jn+`=$R{o3_dgjG zD&JCA+4N@g$Gs8Cfj3?kK%YH-HIynFTT%9ITtR_QuXX~r(;XT;y*H=4cdkuLa4)jc z($c;$P4o22fa~b!cuNfqdZ39cOkU$>+hkoh$~wksbGB@wdauT*l@&2%%@?u})3e*s z((;`pS?lY(ApRX~<#`rU&z7xO_ML0gRo!QFwQCs-L015CBUJ^tIok6D|^7uCV#)mld!Wp*t9=iAhI(z z5WhOdXUzEe1b%;@)wJso#O8*=gw=It6Na{{46gouJ3bpRvX^V+5DWbUsyfPJ!;QzV zn{l+=_SUm4#%mwte1D)Zirr>Uy(1^N@;cjk=G)nFDwnYrER_MSYYE&&-`<%|gs6JX zv?sI3^gAt;R#BEMdwkb?rQaCp(MS3GHAm1(iA#)l2gG}DL9P_1gf;Dko^tcV=PKbY z*4%BCB+*W`f zjr}+<~0sET)UM_k*VcL!tJ`dS+Yt6k?tyO*Uirj#@p4Uco=;6bX zV|Pl6kf#j5F4$hw4&c-l9)ickz)NuL&y*x}wDV-1Q*x51 zv!~7P>ke-acO~=24rz6^kV-8ndv%suEzWk>d&hSp?(^jsRUEeNk69)8cWQSO;#A^L z0?e+DkP)}j#*z0Cn<9dXeCj=l=jFq>$7m=h*%FcPdp^2{oUQ%s&4THbd&E?m>(S@d`*Dk zd2)G(FZk8WDTF;4lhEb&hMR*|E1VP=sm71vS96UF@Ed3QOg%gLVDpU!%o;V{TO~Pd zVNQSD$y<4W6w#ifKrHTW5b0*P7%Bzt`8 zk`!Zy9$pM;xJa^}oq`jGOflD$>D}ytv6CD=SYK3ImkZ@aJ>v^Ab&xt@1 zJfqSq%z9(cPBBel$2((V1Ie`Lh0Y-=+6ap(K&rLrB*Dk8CTQB$VI zJCrC0+JPZ%onl=N=^7tQXc3u;GM`Bd^GwSuViVt8uF@SUaTuj6-;`Mq9@0gQz2u@Z z7n**}!T0`D9uI`J$9a#PD2dI_cE+Nbbo8ap?t>nyR*9<8jUk${?ar+8qXy!F?FN7Y76#qQ7?@LP6bM!#UxCV@SY8Z$E-KD?oYi-svZ$>s{Z2Hz;+Sd8%n> zYv8##ITn)Qf%={%-cJYn=5d>i=Amnjz;e5A_`9VAr9!HCf-DaDj z!2(4=Z%9boAH5}@z65bza4|5Q1*N>3PzS(!mI2m$g^K?v)O2jYbo%2TM?)Dx4==-E z5ifWRaiFHN0H%{K-6;y8dPu%JcZczl>9^rRO{Wh`$JFb+4b-l`!UH>Zz9EhjYC5^n za9AJF-8rZOVITuLCm(!?5$e|uhL1r# zdwB2^Mg~~Mvj^l*W7Yt+r1*P;|G71Yzeo7Lo(tQwe4k7o&^9~`d(WjbzC0_dv*vbe zc|roCwe^--glPJy16j|6Eosz8`irQUPi?Ljpv}})_ZQEs4lSW#r0~qd*5C4&c^x|t z1mwyU7nW<%)3l-ZIUBIoVsf~xh{V0`IMnB?>41~76FVX87@RoN ze^M3riTH%reDkSBJ^Y%i{)}SGe9sI!te@vtRq(!ewtyAab$w zN~EZ$>rNSoUsslnPJ8VdoyzXGlcGVg9@)1b&aMbYN5}WHK3Oaw8*gW?662O!XxpEh zn*rWeRVb%sAxR3mlEcyqvT*~7SmQZfs;wlrv(}3&mb{&2wNd#^9O5 zd3N!eb>74RGgNY|o5Z0;j!822lm`^qqz6=S;!Y8kAIHCgI+b5j=^l5k;*2i{Vi);MDY>c}G%V4uIK$LEpX9$D|!$tpv(Mu}Ke53`**DVKZn{K z?DhQ|gH}53Au_Puqv_Ul#OYV%+p}r9 z+iNrEB#sF>?vk5z36iQA8X-+*a=bRxYv6IOD;l}V&zx6`IWzV(%z-Y-AZTjVNDNAd zD1g|ef`dGyW!nk_nYgNAKa`(cfxKDm^?7Mj&Sbkr?m4O|kOh*Pa z=9e5`ytj6?+d>&$5kPK5u7qFxJ=B5JgU5hHIAq4;lR*ftwmZOf53aicHRcKj@K3DP zbE6N2`h9^&0;^9FKYyS=4)g*E!z%-@h?unGg~Ou#*R%OSJXUam=k@QQ4)zEkj*kmp zy(eMH0wH|kQh^G%5by$O%)^+1XxZ=xE$F3|QV^`;mFRE=1>(OX*+i*AiSn9IwX@kKf z$LVYFPP3T`U4|!rNv4YXSXk5dNn3wQg3AxB_qGs&?; zKuBt~Pf&2E0aKLi;eq%&m>3L!!VeaO?QPac(dC=;-d6ntMA(3N6`?Nhjn2|FtBa+p zeK_Y(Wcc;1c@s}v$IbVWF+yhA>mQCoO~Tg?o_eHh++Qp+e%nuIDExWsaawDo!sSZ* z<4vk~aJ@~3m3kK9@JYP?6m(eFy`<)3??m%9ds|&8%FydsUU+2ZpEKVGeGW3QgyHNl z*z{got5d|Mm!R)HN#aiQUQSU_v2ZfN__CXuoA6wEPG$cPDj&$v2Kzk!(|dp zZ}?aSEE_pTQT=AbX(|~fkh0e-2EdNfMMP`tho0)9B9-ncB$O7#{^59<43&sPpVGK- z_UM@8ZK|<6#CW;U5riXlNVh@Og>`G+G~DWqmMWXT`_qB}9a^7Zz6=oD}DEozqb@8S5>m^p%J* zkp3PmO`4xS#u82;d6N4iQ}NOhQiiuxH{*?ae4wf!s2${a0P705cU# zLKU_9=uAvSgn+T9P{OC{TqFUrf+VM}R~R=aT~1QP5qvVB<`Dj3SUE{P^;jH|cw%$Q zAl<91EJAP|ck^Rg5}G<}^K<#2DfQyEUG=+8O9$Hbw}WRd5wJZ!mB_Z`EgG4(5kqQI zc{$-6$*R|;9!T*sxW>nN@WmRm#&E(+RZ>Q8?u-(pTA2VV^K>r+M^dh*q*FN42Dv*ah8*l6P^fG4oraSk!&}cxms0Y8H#h7&UU1B z+BjufVI!?4?VsNu$R`9K1(UwqR7Xd>0aaN?HJ5~S8l5wUhJx?f$uE`LZP%E}HtyWy ziUf}IV9xcv4}+uWIhNZ_Q9`b+^_6pj?t<3O=ktubI^6|{ju&}2j7-tlJ&2&;Xu^AZ7PL=e&l zGA)Uz1Fs)Fl~{n1g>5I*&t;X<(YbDpe(yalwfVW=da1~Hx-Ewh#BsqW^w36`kjTlk zUV~VL%HY@JENI=^J9XEp{N8YLAU4P7_Hq64_&~x~$;q#5^SHjMs%nC|AJ-@NE{sH4 zDeD6BFsbLXD?u(B$lcy%*3M#mDZLvN=+Rx=LVIc6%*)>-H+9m5;aJUrrO(dwGY>e& zY(1$T8z4pbgHfmW7-D~5{X*X12a0I*kEQ?$q5_9r#=MmnEq~5Xm>{y0PbS1Gp zJP~-+-NC0_Ctaj0!UBZSoIyU9rQ6nVqj;DWaB2CYJTp;K0DZWcR7V+H7ty(QfJ|Mmu66LSzY`0xuc0$B`eoI?qZ+?JJux^o1^8^s*hlO#Rr z2>(S3vEmbrzihngEWrGvWp)F(;=s!NmZja6S3>ln>mABf(!>(vbS`UO%_rX&c*?w_ zV3typ+Osa)7;*|)$KZ){(YsrT{co(pf9Z_4Qta_Z^SwiaYQLf-aqhtUqm|LhdwlnH z-zTdnBK0x*;OS0Po#?1zZjGD;&+R)5c3)Is#Au7u{%*ECa_nrk-Z2Op6Q_@h0XC?6 zCg8iTR240Nzl2APxzor7ulMAqblwv<8-0dq=FiS7do!r zRV!BF7w+G?>J>YL$lAawt?!1 z7K~ur5M5BlZu^Q;!1;5Ear#(7mma@qf|@YycNq|O2)k|snHJ^*!3@pwIls2<56jsB zWV9v|Z}(mMhtyzOsIW`hM_|C~^Wy)eh2KW$D@x!m zXtO0krjv6)oX7Nk^Ah1Q_rI0xC#hE)xCH(@QsOq8ZiI0ZA_ zJP<0^F&PNc8Jnp8Bmy`REUzZ(10BRkg2%yfa}&EQ{vp9ZY(V#fZKoCg*#=%fX^r<>T>hC|A6whtAg!pF%fMsyk0E3!jBNrA8qptqZKVFGF?=`taPCNv1DeqPO`Koa3IvZry9w@iNU4mzrGYnR9aym5<(+sw<(ThTqS{(cTGg z!%L$t(#sX>HWwp94RFoxlxLX=590hYaTh?zs9udj zZ-H&B(~4TIROOKtuo~0pI`e5&SF8 z_D*Fji<@v#>FmM+ArH>NT(4l#LiRtz@#6uP%})rFx3)b&D7`x&-H)0CqO~sBcStYEOeDf+S)!uQCMgiS5 zo4dJX4a||wDQSt6l8x%Ev0*$5haWKww}4-nbD@_OIF%#ajIqWGd;O}x|gAXeXa?}g#KE@s0l?Me;lD$gwV*=^w^yO!9v1kv8o30$V+Mc z-m3*g@;=R`h3`3XjYhO%N?q8f^HwBp8~4qAB*fRheD)~Yj?dn^R}%^u%6VB*sE`gd z{L;7Q=2@`Y2o~RBQvS%gSN@pDMxYtnqn4;QEhjP(UtwpJqaAz7T%5iY0BpVQre@;h zvFQ#N!v0Fl_WInj?qDpAH%{LMCGzPl#$G$*x!t=G*V2;7-L>GeFg?AwaO%}qn1s5a za(ATMVl8UH@c#L}4)q-3y!H!=EEaArYH;OEI-ml|`+#MDU69sh2;D zrJdY=9H3^jzUnHjVGZJQ^`S}fZ%&Q{Am9>+O~;klv@B0d%mbJ-U6D1y*!8!*gUHl% zrm?xNnZ=9OJ>b%aFdbJB`3U|C4q8US%0%aPQDo*s)M-#|EG~&5aBTW~LLGivCBL1D zFCicetvc~d?H@z^u>FXm0UVqBxBA*p-&rz)bn@L#{Gor`@zYpFr~t<%!|+WQ)OYvr zaN$;WFZwb6GRS`WHSoAo!2D*{n1uR{K^54a_d;&yK`rC&691m!|H*{f7M<4hnB!+K z{?!FUGuJL}$y)P%&+LS$D3nRX<^H zuh-C1<+p@jc*O~vBZD+Ljh{~Ufj|7?0`3F2lAI?463%M^SUu-1amHB)R%eg_tnL%8 z0tr~o2lAu$y;s+vSsV=U>Mf91GV*CqcwPogFdLIjY2!a9`BMyC1Xs)+WL&?b7AkmuSZFCoE(1$m`Y%&}T386A5 z7Xbc8xjL5(YS51u!K)&X>j@=X$w3_A!UK&{P=oe@#LA<;$N2Xc4_N8<#s3-HcvS+n zH>+AseX(%l>2fuhB0=#R)q55)!l2x$YD#kBlP;H@EELP(5-`7VggIhrTDe?iZj6HL z3GL(l&KZ!G(6Qef_0l|{kcRK{dp8dSZxF|@2MKEf8m5PahO!^j_$;r+?u+*heXkFa z(=R>%$Q&|(+~FcPwrntonRh{z8)X}%Shs(`S>2YO(Gx^RHxqOFt)kY~?}kZzeCGJR0zpl7>G|u8BNO9P%DEB&wb4nhIn~Jv6_6{I~zJVPPO$0wK zw7b@>NbE)LY9B7;h90^0U(EW>(c8puMX`n4TMO=fP!b;CMB%eoh6|qBF6*7k!~1)~ zx}eB2J~=tr_V!@il6{?6Wr@S;>`lHv2)T-dopBtP`^J|7b;OmkooPXgs+1gRBsYaepSN{(3f$LL{Nu>}A-B^s zh>jy|01I0}?ONO3yFcHG=J-*X2y$Qqpd4JmNksUeiKx|evRK#c8Ky5L`9-Hqy7Emy zGG=tZs+EwdZ1cg#9#Gg$QMNOZ0u+S)J~sy01R8WbOifO!e)b1!0$}@Iu%Z3Y>8qHBX`W>}! z04P^Vl05vRg{mr43iEnR-x}8MfQ9vJW8Tl?@^6(wO34F}o+< zI&9{;JnWw($X1O9NX5weB$s#7fHGRIIy$HHp@-v;XD)-d18QU!{F4JQDeh5EnkTsQ z=w&at59}Fcvs+WK#6WrwTTM3M6%1J3DBMH!SG~WRcG>$T;j=m=Y)7)gy*b{FsdmwSd`ciHz+2b=rV%e(s4 z`e{o$TXz83p-{q&CHJj%nFISi8lYSPcM0!EVoA{X!<`A=6Cf>mrMgu(b~Cw1hULb0 zlqgU^_96}8y?zfdg1?$Yl9UdVv@S<4IkQB$p|?hJV)slQWqGLogOR|=&w12*0{Q4F za=LSq%*=MJJr>P>&$6ERZvXYq?3WB|DkF()d-cP<-_kGp-0rn1zNm)0dPlklTfTjr zW2H~fq(|xd;*92XY<*XS7xbbaN8W=(u)5X<=be3NIy?M&ML%BaSrv$2q-JG3qVIc6 zsvobg_Y5^>TabEEBX`B@LcWo?n5KFe?P~2_+Mr66k==r8Oo(7-QS~!!*=@;_N0iP^ zo1>muw&s9LG>IFg!rePd4p|3}djoGueO&HqENCW{tN`$rVgR`yvm0gAa_tHE8m91n zRC9P+0uhJ!5uloEazT$dW zv;XkBn)t7T{z*5ZBjh*YnRJ(>Ae5yZ+pBHTxY+HE{UuO6+%DT|+1=VscOxkjrF`tH0^|_23~Q0YyN>`+fKhvFG#?q6D~n~I z4^X1xX>-Qwm+s|TG8v^cM(k%`f6zwKY+M)K>k{!jyIecd6vD_iR>u3SYF3>>hoP*c zjy|j1tjn>^=}`&ZgD%exPn5+YvQW6W)9qj15T^CDr;i&Cs%gqZwzNdJq%b~AP{5po zNX;J8BG(6syu23|r3pVM?1`Zpzme>;T})HiRyTfwISgMKq0GdM*Y+I>6L4SqwHv!U zBq=N;I*Mbx^2nvG!dNOyy?b(LU88HwSxg)VQ+`jU6cTtr}M?xTb@m{lBErT%n^f%I! zGcDzO0@sq6y8`tQ*Oln{g#lJZeis?4pT%3i>3NC=wjQxktDVi5*6^M3nAqwltKXj3 zBFnQQZ7u#SCTa_~DUZEAeu?c!QhL+X;nUS9m5xyqTa znI-yOn{3Y<^k|E6|6)&)tvX?Wghg53U`h14afU+q;8(d7hjKyKI$ej4^QTEao#{Fl z|FKi*xa?ZEcLcy@6 znvTOn$isW$%^xhATLI}wP!(8CG<@%idq(LqKHfI}<=f{*`$Ng)12qqK!l^|^y+N%R zs~KyRIXe_?O-5EJ2o2t`Lm#Ji=B9FMyL9|+a-Fef?b(ZB_TI_dgFt}SPd5;T9BUni z?n1xTG13sxby)OS|1+BsO6ie+c-OuRCU}< zQ}M8`+X4LoOe*-H)w>I#1Bm+1BOroZC`Q4)ASs&;+Gq55vpGLEM$c#NOXb>G3mCpqX zXmeow2BaWB@K;^!R!wk!H>$HnthG*gpjCL>cX>`{UsS}= z27mx{QzAt;?-g>3$VkSlph2GmPkXC(5*q`2*0!PB@3Bp@!U!Rv2!=Ai7Fz@Tdme7K zc}aF;6PA{iVt|qu)1Gi2y|f>*i>{K|tuo(R@Ak4USpPs>R9svIik|{wRrr1>wE6XD2D+Zgvi9fC6K!>-`edj=UhAI zv6$DbJ=$lSMTmSd4~eY7eaFn5gk?Bi4TNHqMICL}+SJbQzN^OGN9e(y^AN-J2#3}) z?U)mRtT`V1{DAZiU!>waJO?WUO)CP4{Ah$O=J~6W9-R8&i&}Beng)hpl-n<)iA1i* zKx_l38j66=+9CE^?8?YgGVk@~A|xavFb4N1Kt2)L4ce&knqdHUz+#KMXaIp`1y9r< z#i;GR?y}H<-qH}+Y6oDTjxFz$@9$(>eB=tbkNo7|7SqDSV`LPdZAA+p0}L2SFuckT z^5?$t7%I>~Pv}%mHb;7yD3pw10_@bFmQ4J6$)m$w*}+a#T*WYe-LT8X>)a|{C>kKS zs)ZqfTlhjIJtQ#>lW(|nE1#1{e*-y&=l0`_M-Ff1!QS;TxAkV8s#}*tZ;oU9Gk6bh zlqz$L+AE(P#jivhr#lVO7F3T~V`wp*LYBWU2IS`8bsg>=gy5k(Ej4E+JK$;Gn+=pJ z92U0$bSD2|^=V%Ls@EzWO}&MLivYH_GRt*6t$-v;BaiI|P#6QH1l;S| zX#Gcj2c^@9me>0;r&^=t20l3NT2N@|qHt+3yo$I*ZrFHXId*T}AT9op^E1iWl!VIN zt<|csokbd8B9g}dGiw=`Ssdc19=-UJp#Xy$Gs;N6^@Y`TFzI0J;DUP_+oiw26Cg)C z%SX|da+2(sKUlOI5Nmu&{;|bz_SmV%;{qhN4ZX1p&k#xNE-z1TxrvCJ%%vNQwi}{C zZ)fd`ito(ljFzmn-~+UsrKQ29<0|!UY#EsCIo$O9nj@pCI{U@Cc8Q>hbPa_%ziMY= z!z}ld$Z7QKBk|>k9qFk;q;!)$?~haZQc&@wwaTz<3Dh)R#Gr-e%lFY3*ePLEYvJY& z(Agx?ZT7U0_CfC=&B{J~*6@SssNxQv!^|F;ixTEg|M*Z%&Wg^u?b7IcPV3FkPZ)UK zFl1*T7=j<;HjEolDT*Vne9HuGhIP+ES`$qc;bTbBglk}9fVr8^2M5jWsbF$?RZsLv zcy6Bo&X(l7eiKwVAq)}!;~})e!&@zO=lg*#cigMb)@J(%vG^--^dRteXN8G>92K16 z=y`<*a!mTHCdr8G4>Il`y($3no0FQ)t_p0v2dre>iG~mxK@sx~x7|9oJ7%uA=A)Ge zGKrsFBo&#r-7u}x(4Zxs3Ptl_sSh$+K-w$1ZMW+J;H~aLU*0GnbpoLwdp+;&W3=Mw zI^~l0T(>4h717@nG1zc#Iky3Um_Tx{Z8MSva@XHoV&G2J)2Bs=m zB#|$c({%ROK2`0=P->A+wnWolfTh_0ZM_U!VGn~Chx2zb_e4O1!@ONVNaXEgMpP|E zjZ~b*+nuEe^H?u0+~?F>nwwKB>jA=Yy`ZhYgxx>fW(noy184{Dk%SOycPk2Er$6Ae{6*;c-FPzBbgt-8nRf*xX*SL6ViOOS%^5il?+mu&HIA*)-(_j!MRiEIW3sbOVcYr zAJYj{I=eMGOsg*YJHzyru6(~X=P*gY?-ycZ2e5+bXXY)&lcR0>%=<}(R%14pR?#bA zR)gDq4(R-Xv-P45Q9k5HV

jRlbL8$}vM+EZx zRSd2;Xp4eVga7;PpPCyHjAMJI`oT|;DMNX6Z*Xy(ejga}AG3fo;Q$T1s*5WzMdnA& z?Ck6W8GC4(e?IynMh6N3M_)-38O;>MA5oe9v&JNsqwQ;N=L#>y&v6~dfaM`E;(nwO z6A5J|!W6g1ZZ#QHg3e;3z+5|?i`Kdw0{$Vu|GU13^ifYLbO>&sRNdMrJ(Iv-+cINH;i_HsE6+0 z$PZsogLEQI{b@xAzXZMr5?`OV;)$S$yX`8#KU}S~hoVJG08?*v!LxxF6NZol*+g01 zTBtbrD<_aJT)dG2+1ivJWZDD+d14NX#@`o+0RZo{(z#6g^I-n8ALCvFsPJoFVI3rj+=%ffGD8B&gD*py^2h^CLbms37{yoC~VAB6KT!?mavco>Fu1oq9}^AV<|D>A{Cm@Sz}xx$#VYX}9#OP}yuk+G6Yw<=|$B^d3^T4yK$)u#=|%3V-~7BmDbaWK&+->n0tJ#>*<>W*#LpA60Tr*Fcstc9so zp%~cE3F~NEz9O_N<&JX4o;K&5gp!GEDC|Mg;b&)T?{{Zv=>kNf$h^97QAc|_88A{S zL8}~1GWXJa54iZvZ$ENxpmSC9ng#;`19ii6-jx_CE)WW)rwk!r&((4Qd|}m6c64L2h(yW;*uUvbivTuYXkzgP@uqWF4zcl|u_3eMH7mnqf^<4=^+ z)P7w0kg04YuV?C(J{Ll-jB&bF)Lqm2DO^fBo?udPz}8?B^+FRfDxv94&JmZf6CcsH zJx4OEYCGRUYP$s7nb)9Z$}=-FvuI2SmhsYbOz5_=Q3+3bZRBKLjeWYX_DaOy>mBAL z#vQfmKcspD=ZKxv-q`$XLE}znNS*A2_0Ue1+d5wEj>TJ;hRzU+bs9)PJ>P6?JB?_L z6k;(5Y>JgC=OKK|9EOd7W;&tfuWbv)l(S8oOpc2NGzLmzrQH;2%fsvV^4@bs;=XTb zK@@jZ6Vo>LEnRh)nWdf&^ay7gH?@&Rp1vSee6F^;w>RX)u8sA><#0B!mYsd<5WM7; zkFonNc6)COE06UOXQ-sbaHt%8@bCO1XrvBcF-eb_mG|xn1(`I}r78Cf@1$C$x-_BzLrliFX zQy&(yPGwAt+zCKbm}3J|Sj}6tp=Kk*mhSwdkb&4sZ^#7Q04}ThD^qwP zp!-B`BPeFFWQ>DFMBbVG)oS7otsjDYA}-`y5)?~(@dIN%xt+>*h`$bm*qsxs?kYav zP@X@75E%2*wM74eA0N6;cd7O9p zhlxNo&(0on8>t$rd0BPoI4${203^%?opR2;=m+INAlZ%Y*~ArV@$DLm2LMzSI)@&v z*FA`%MsO%?Vo!rtYfysHE=r73P*ha(BYy|53e-=)djXdBmfWO%q~O#5)+jD_0Q`9L zW8*~{{w&J}AY%dyP*x}dlwUor?PwbSK^ETGb>5rPq1OgDRxgZepjB+^@$zC3>(<5# zMdx$kbLy=pZiHRrOq^^HTavE@3A1TT88<}W@>9=|NH{q;Re?iI;}CdR(T?r3vB-G~gZQhCHIhc+fOK)x?j?4Vc$pdS zUQ=bk3wL-mOx25SGV17^nIYL2Oy5h|Nk{edi!HC73^O>J)F7-$tLzvRT-VvJx);u- zlT0-}@j-twe0vWx1l?<`wHQ)GImJ`moRFH+lOP(i(59PGD==PjoT%$CUyAk{nlfAn zsp|k9EQR7d@Kaa1WMhX|wk1XWi?RNu0*_OelVlgHa&397HU3O3rD?%Fd zG@ryJ1jnRZ$2S;`ARc3b@sYwvtxpkw4ykZwo!(L>yYv1*AWz+~r&S6Fj{i-kcKPlE zmzNivBkPZt^}C>(Y#D(=`+P(6YnAw11Q^sOQuitb2)(j|fmLqSaJiBWfow)-eyR;q z8CDIpZ~O;rd78O5Q(a?T8qz#RLHAso@1Y{M+8Up-6wR4Dldpz znpZ}iM{pRhXnv>(DtA_GQMvi?v;v2Fty=~isL4!%^?hM-p3@vYfz%N{G7q@cEgZZG zCUt8K9Sa!udJIE3z!p7J-}N}%dJJU0rD!q>$9%FK*JqdCQkJIVMw3Dq?>G@$BLrN0Nr#iiH-GU8?Nq{o-Mfiv^7>K%j2U9D8^@gNMXPUc zkE>r*KQp!zyXotLNg!^PeuKwFz_-+6k-{(PUSa|QplkEZ%)rLV9qnCOwI%Llg{&b8 z%a2+m^SflRn^!i!2Dk?Az7+;}e+&>t=$+jMZ`&SSFTyvmEdVP`7RYyL3E-r@<6BD9 zztE{@EPtMC>Xf0#pdE8^cqC`JN<)X(q>})l#;S-PhfH7MaK$q8eC9%cYmIXInu?-} z(e_j|?XBqYqnVpNy~#{pzAR{5&bS^Yo>Zs}f>6E%APDt9MjtCTM=C7CpWWg;>wCc5 zAB!%#dG`vpuxqg;vj==~Om5!@-dxo(zQtbK(Ji!0cgu`md@xgdalS;6aE(ojhmgGO zfpy*)9Y zYKsE!riaCwo+t6yT13-dewF=0fBy{dQ=bD`%y4(EryqoCFd#3lBSgXs&a=v~7|lXU zu>qiC3NhQRm`VU3hY8h}kahHmlQoy$k4=%#@f}hHzoIhycjMN(oA~pJpV?7xDZErP z4yb7bih#*^=7tXShfS)2GLU?$5-LObEw5*}xtQyf#}*-LX@Wjf5lfZKTJk|fWDH~A z@%V9|3fAEVsJYSuzY1VWBE@?Q`M1Gab870@y6;&j+!p54+3(UcBLw1|WG!v_b6yQKK6M4S{*Dm%8h0YO-hKlcEcsbf)LDH;T;H zzRI!e|9I8eLSFazxpb{!L6HROo@1Q9R(&Ya1CP6a?ZN3@o19zcMV~0Wy*6cCv@0Na zNt||!?Aw^j<#GLGj4JXraH(~(@ZKQu+aS{_b+joeQiT#D8mvk^*wf%eV$@FxM?H(C zlK~$0-WY)NCws}!IWS>5)M2z_QE0+|YM+Pz4Q64SuR>BmlA}$4+2E9Jkma|pBxrAPO7H}J8>Ok?w z&4}6)lq zrJAmPMI356{PG}(uNzDaWvMb414~uC(c2zsI)>q3I#sL!;1mz7#DP5Ab@{c!e5fRt zNgOx0-{V@IKSb)7H3n{nEK0BYb53!Rh(Seh3P(HBEVJ=rPdWu`9)cng1s#pf7$E?~d*e&fEYf`*1#kn*d%$_pylps9abF!8W zi3$>iwovTBoJryrKU(<5O1gY;##r)7)>sDa6nnUBZV_(=TFScHm53$G>O|kl!CCQw z&JcM5I@&t8lx1P4R7`y9dr)&<-d+cT5<2Dw5~15&wc_qL@*@cJkOYLN5Cb%^BDz9Z z@yv<1^%R9=K_r(Aa1SF!O39PAHmj)Jw)i-2w#>~KiOYT?et}fG1UL{{_yQK z->n6ImQ6a5k-PGiz4YMJ4F=H3~R~4bE)PrmK9OOsB{HK0`m$K_8(v)Z+BQ zZMnV*03#wkCveA*hQyo-`MlhB@!dbbh-k5bg$I=8t2mfKOg8tH7>VM=ay9|3G3SNU zoHRbNNhH3JsI{%_9lr|{4dcga?Tys=JXadK$0%Ps0?~;w8(X49r(bTp9uud^yALu2 zDs`vyQg3Wwqc{;y3Yeh2ggzSlTbzWr@q~7#?RTA*%^ZQ)`E|*vz z@aA45@9Vnyt5HY^2HFS6cDw95jG9@<`2qV*%`*=cF2#HdiA|RwYu$|kaj0+sW0kJU zX%}K9hsob1EvG}g$N_kL+aRt!SCGSe$ZBN$H;FYZaru zj}CQZY}U4lkNeQ`-hb}EL*U5pv%euZj)nb+tY>R_aP-5Ht0384MDe!UbKo?+xQF}m zO_??b%@i1n`_#27m+t)1z!B`gLz;B#r}?$bhsRa`ZjkA@d@ZQWX}bs3@qgpSIDKqd zTF(AF+CQOL2o)e?^KTEA0newv>3m2j#=iJ3KD32u+Iv7Vlf2!5f4^JiXX>l7Hx?8!Q&->x(c>dElQ#r-icuO_n^ zUupEBSG-kil*)XcjdA-U$Yr*AEGHr z5)sQwyarx>z)1l_lm+(uzd-HZ72*sUYnQ)sxkkNs!s*aiygd$>{|Yb8r8pVD(PpG2 zgHXbm8wsR3nVAXkA({M3QsU-b1$-;B{4;g@uOC!Y0|j)muEn^w0sGGyjIinL)J8L0y$#(|JoumPEOz1 zHau0jck+vmyB#9^;pH>`nlZfUXi-x06Z=SNCU{&aSXDo}&RYn%36DF5SfuXCWBd13 z#c>`5nfvFQ`wm*?AH<@YLUg}HdA#wKA;Ixs0C>gc9CM6+-ojc4U&>qc{89z;vdou{k`NTYngUe*C9n~r z^+9pC*mO_t{G4FI366Wyc)khe)c=(s&W(5HgZFbAU50{d-Z#N+fzd_=<>BVa-=9j@ zc-~fApueMNHy~kWF&)Fh65TXl+#2c^Z!kw@Q`gxolx-weS7D!_2bk`8Jw7VUhHvgq z<%@+)#ssMDQg#G$8!N&|YBi~?*vcs(BCR-@d4Dyl>M3Q6#s-S4bhLJki;%{3Ew9L~ z9Rrtk)j7JbxJX26qjJgMOpz#u=1yp50&o!U+tZFJk;y^|4WM`OObsmMI9tYLKsp?wn~WP~()^Sef^O@g&-@m?(@A>OIsu^$h`@XOH+FsA= zqX%ui=xeoLp+{&I1u0{(H$#lRJCN>A)b=VIRo8wdHYFv+{nt9XINmNDNuKD&pP!~t zktVt;*nrJO)aAaT2yL$D=KGy{HlJ}*|1zGFk$?PQ2^r7HAy>HFV;AgJGl7?6X1Su1 zA(TC#oQZ+;jh}|K_}XMFPyghXC@IjSEqi=kB%MF8{p7-$b!nO2#MbSb(`M6FB^n+z6Hb1)C{^D=MPz6e;;_W#ij zq8C`wP$2p^45TC^0=GrW#k|4~N^J>m_0 z$+6PWLzoMgpMs?RDo~7`e_Tt|9HC$Do}DceuTUvX9?ey0L7G?ushF8eh`ha%f2Gm8 z48UolmTf{mFYvluXxS;h5Z&M?uwr4%>|VvC zKbR2l>>+gD$V5R=wRdGSg9-60oEluabjo~GJQlTUN znaNPsTQL@i`S}U4ia+yljzT7Wo9D;J4_BC*B|UI`z<7-FPCZiNBZa3o(nA2FL-?>q zCBp)UBi9*OZ&%llVALT0Hg6^`Jr!@4Im`*(6W8O6qKQH?e~|YdfkC?W8e66>E|~_( z3XMVD;QfB;4KpoZR9lAd0E0SU@V=%eg&`BDQysU9(%MaCe}3T`SEuDmc;*%#-pP4R zZ%{nwc1cDP6a|t;&b_Mb%*JVd$u#|N=w@zDUwwT&Vr3kdN~Y!GVxtUGZk-bbDag|` z2M33M0^)@JYy?}gkLR5D*igi!tLC57K-R3_dagN*D_qJ|*{XFmhVt=f|C`V8!@7?( z3>G?@(s6tXA70#TPi=BY6GR1b?r}ZNy_Cm!`CtjW|ob@DK8^cMae_UP3-W*1lW`)ORm z5wcS;M<>5fI4u+xAEoV#&*@24XA*nd;8vgcA_eOWohYJ)S9k8?o6U7&(+UdC@yz5C zA6_wa9*ygNt#7Nt&~dw|)BW77y6bCxN7uf-T3<_Y zt)O~QjB}O53)7}%LRwS5`ToG*CPr`0;TX(RwzM+=XZ9^@gz|uJis!NQavOI-Btv+Y z#J#J@(c6TJrN_jITFtHg*cuoIO&zCQQZ)=6l&pg8272g+Wl>5ezc>9V6r^Ev-^nVd zIW(j86j_c$)QK;TTUM9ctrL*w`mOjtYFLMAcZft@KFk<9?@=4KX1g#Nz=?Tkpxu() z6Gzs~y_H%fDR_GxJGfw9iF7`OQpBL-maOfh>p;TwtMQ6eL>cxV@lWz~3z^4A26 zcjS8`#)D`|-e)<;Ywe_GXIwpLlk#=jUn4p;?BWjQssbg^S2zWKS~=9CFJg)Z!o+K+Uf zOkPL4evRQa--}OQoal-i6XX+_{{F7}#RZAaMHK4}-{gpM^Ulei?Ivlb-9^djl&_cH zJ+G|SH1$hkWWQp%DC2U|gsbh42cJ)V&hW^{&gn%Lx4Lr^CIeRFEDoIjagjUmS}dG8 zFgtICb|I5IVWp~(NzC1)A8@#mP*E`Q_;3l~8NPv|!SV6O&K7PR^AnGw>P<8<@mAOL zor8V}Ivn_+pbtjMoi=v6VfzlJR*_eUiPOq_oT}$*WvAM|7&DjWIxTY;UwYU2`SZ?- zs^+jhC*8h}CK{iFdAam;+QHo02fTODOYxp|)4sAPO~Mrx%L(7mJCXRLK4|e^lB18& z(>|<<>DE5l2TOx5f9GfAW(n8LI`9S>N#gW&n?_)8frLLq4+@)RzVE}W?vcZjR|8H9 zMTycqqhv~XB>Uuh0&Yu)*?;e}f;oW9#6)y2`PUTN&^=>U>gT`gmR_gpk;SShYEuC{ zH)r;KPz#X^PSa~@$d-7mciukLOa!a%FU005LO-bQe@ArsPrP5V-qnU)xnTv-=94&c zcGKMl86T(SU709f@0|T-8~v!Xnj)QEq>fcug+IG+|6=D|Z7aj&vvQO-<%}YPTmPTN z5-_bkCEzTX`KM%*xKxEq?@73s{#Ga)b&&0=sr&KG|Gv*+-@;{h%q0Vc0f-vpiFP8g z9~gh0oO#qF^9IH!P6goP1);Ihx>1tl-=F;v2}&82sywll3ObHg01!bL0+ynlRF9Qr zg4bRV-F038XAqOD49p*+Rftcmysk%8XtkYAW=jrZ6?@6lqh1VQSp!HW1 zj^}4O8YUI_%VpnLgV84GqHHU}`-0pUPvvvRZ2GLqQQGk1L zZ1B`6stbo=Pvs{O|8=RuHwn1HYiXs1FP+G30LSn4iBuNfl+fA~=kxovov&A~DTv z{U1qyADcM5=3?XDl`d^7{Lu3dt!iTl zP0{`Fod}Ru05rTZ=qoPs+be>QujRhNF&rQcXwNs@FRTnBuh1Q-Zr=&Ssf6@a_%k1 zg9X)Fj~DwTz})d)4!`%FiR?uwzaxl@fn z_a3;aFOwK;lpiSdZfz20%3L3H%$`n>(|DldPBY!qEbi7z-|N~#GHg-;j_%O-IB%W= z!PqN46Vg*X^zS=d-DiRcyO5NbV_f0j?LJI^O5#tcDV42&m@72vhKA)Uzj!2wRH*j` z$$kra1EPYs9TD}5pbUqSRNtWK9T`A;t6r^>gcaPi$Ei88`xz@1M*qZg5+-ARW8}&y zwlBklo!i~Kb;mRMRhawCIO)<_^t2CG+%(=b>EXEC_qt!P{F1o@1_zs2Ry}C~635b> zCngF6^(8zGB}u=hvE%{VbDp(-9hCD*1w5UqM*x`@+-Y=kd1kOWOUxiIz=?^d%n%0G z_#2vOj~H>+wwkiml7-wUaMjH&(GrYR4`$JLo}@?UWn<#n0{x`>q$}#gs(C>`e6k@VdY+)5e*4yT5noi;d{h(W3%OZaC@{!|^OZ2q& zoa@$SlAlsSGp<#jX~Ks|#SeGA6Qn`=6yFS>Zu4a!G}rIRjz2H125`p_G~K-JY}zew zUh_t8*(1{<9+l(nUdaiThJ4$iHb;C&9RWa^;Li87^eCXw#}&}>)J4f+^h+QH+TD?b zpnMku-Z|>JY3t|TzK#+kG|jE8p-~JgFg08H+MSBM#s~v;THNJARxJgr$@!EA4M%hR z^a-0hKW%mM;?%5~DMl*zgTX7x4@$$f^#wEcH}^P>eUTVjTW%`*4vCTvo#}h&m=C3w z!ZhocYxdGWTN)@aNIoUcG6y?et)Av_XoZp6bj8lt!-3FgY|S>snP4Hw4ZFk-e%7l^ z`cd5I#jWe?mBe7{KasKND$E_R7+wX9$ffJIg}~~4+}0P4u-;!{`*6!MXu@)MF)-~^ zg>)q#_k8pztdQWaUWo8!ZO;ZB>G?;SJSFJg3u6dq-ZcOR8G&O#zdrgK-9&)R zjS0{O9#s@jD)*)1;9f1Yqya*jb-{2#WEt>>%c=CbM`l2Rj>o=g->YNh2!E@}3ir_| zKFCQWT`0wcas83nokzp#zNXTlDsBfyJ3vQ(O~hpLT;>NpkzCe>UGA+#-FpM&Mv|K3 z5l?|b^Tzl3=<2^(;F(UG^Nv>!^+JQp_r>Iz|9GV7phMFdGx#NM{0_k`Ke%B&9S{>% z64-eS!1*v&p zXWgIX?N^EgU77q$m7wwLw>o36c*bxmokzRJm`1T&gCot0j#UeOp|bem^Srv7#4Z)y zqcZGDDRXO7-=$QyQ7T0z&y(;NY`=8L{a0E745}U&<%*B_-PDRB&%ph9(Ug`=#f$Pg z28o|r6N}V4E)AVpMGC;1;!iJ%(J3m+ow=lO!xaM6$z< zzvHD?K2KXbsiU`1g^2+R+1imd`Q{FJ#8E|e1_&)R6Z%C$J=xc1M= zR_2EKWv6c#g1)t5!S5&^zPi#% z)FM}`9f2$b=u7tx_ka+G_SOOf`wzbm3NmC)wGQp^M$Rexw4^`&;jVhD zMDafGPjv6;wPTK5oTQs}^>Zu5*tv=bd6vcNfcq?;z?FM3xTBn(5k=vIw+(m4D1OKJ z9yHy=l0k@Ob|to?h4E^#h$%td*}mry zyJ473KCU6Agt%Hm6wM7y=WboDI-@9=G(O`?nj~U_Gat=)TqT0c&@UR-X|O|cfJ;B0+VfJkMwe1G1wO|#58k`{2pnw9-+1XJEV4>aw5om;&W z8;3IgZ*47Y93k7-d&}0b&S0+0E8XY0Yzr=Ob1v{un(7L5kPR4-)mBkRnu;-A!Es*hv0&-Pb~& zF!>3@bS;C68Rwwd5A;qM%jNu%`Lw1iv zcROz7Gw7261|DY%ImK!@xDG}~#RZtIeowP?znPUvIXv~-w?vlyT_H(99?`n`qHON0Ads8a>5Xq?cgQa}A z6l0|Ht=Dhy{kh`w6^AwqVJO@`Z#7c1Hsa<8VP^5c5|Fw-<*lqCnkr|sGt-A>PH}oW zagxOG%Qa2b!{(!FIE+`#k)l*7EU#}zB`sqGEvx%%WRC{!_}siL`J3(bz0%E<{pPE~ zjd*DbIk}-@?hl%P78!Akk#NMVB|BzsI%xgY(anG)T|F)5T6MF~{Tkx#?yK>R{IrAA z+b&J`Yj41m`)!TIW*4y=MpJB$kM_l$+?!Xj7O_jeVb9i^z#-BlvD5!L%uHB_VH{kn zTb;#HW(O3**Fv*}A2pHHGoOC8ew&DEzb%$Atv3JJ3#wHi7EFtAi{QXMYZ)e*5vBMV z+{W=vI`+Dw;PRcD?LRAOSSj4KXmk>rl+@AOjIn(8yAYQ#_97`{W2I&Upa||<^-^)e z&R;#m%s&v!I4wE3WGkxTym>C^Rp7b${c9;e_GPAjaJJ&I*@7v;5k!SvWY33`R7${)wb3`qrbZy+SMRy8l81e_s1psKxz9nMwzbVi)(t_W1}t#ZNz7S^AokfnOV~E`MeHnE3lb?nKAR#$mj_8(jHN z4d396y>!77(%#SZ`)ey_6hk@7m_cybrZ)85n0YmEJq3VL1TtW#gIke?@~wQyjN0rm zsh*t6*xB%gXMO}cWf4!1uPbSONXQ!S?p>{lSILpXTY}Hg8_4gOT;}v%IvR(rmujb6N(IKC`Nhn>vsRSX^S6o@ zh85p!_!={!;3$0`8`mlL$^ZL;!9|7f%D9%xi6}@FN!xU6@ORA$T@e_?J-9BcBfth;nDAK9|#*mP+|9nikau30hpMU5X zBVhE-`c-634pww&MoP!09LNCkFQSPb5ag#rmfBwAfx|ba@#g!*f3NE=u&%AOBkX@! z(pX;_46a^-uP@tSxQSWZ$_N7bFa_89_8~Y?hsr*jXlKQGh6&BY*?!s)>2z~;aEaT3 zD^RZDR@!jk4Da3$QT;s#QO!)?BEkp;A#I-^oMHk5(Nc+d5T<816>dFgqyI z$EDP3k80DH zZh9$2*2C{?NRWNn3E4{Jr9}KnBC~(U^YwDjKtPp~QH_)80LZp^S~83X*W?r-&lC5? z8R50(Qa{d&BUrbW-XT$ZmfOm}(#(8>0-t{YWxFBIN_^YoRBRKf$F9qofwtATHt5a*ty|&-`CK)!lPFTrE$Gpq08fFI) z?ZxDA^gr#-svo?}TlIN3a+qmSC4T&IIw3^rL*AVaTtR1+TUWK;|KQ&>Q!Jlb8d>B} z-h^ElN00(02BE~kJ+F!+O`-=+b4VV|lESx-=GKqb)F``G`M2)%ZU=hXBS}1SIheTDa?dTUN}l-=%BbmozGLDf#h7acbgn;K_2TEuvnQIaHe5=NzkRa3 zJctG_xH{la(;uW8kt^$eaEP;r5_8TidgK{zvfdch{W((FKDU;aX;W;!YyKpgS!vlH zZoB8zYpnEOWkQqgXS>S|w{!H3691l~p{Hf^^XGs04mZdb|C86Ez#Sej ziv518KeCZM8I^n#epQmV8%UiSpR{nOJ^I-F)yd2+7x%Qupp-pMlV03W+_kJeqbPfm86zb1NDuF7#ue^rapp(Cc7oO4n6&DHZ1LC;b!3~0!`L_+~GsI;6Oxv%-%h2 zA9EL*{EKmqluyFzMcx{d|~`%rQ7Sjx>%?CXFTOkgU&i< zO7RTPQR_nn)cTojc5puY7lArev~Is4Zn* zevBQS?JG=9@teza#u|UW&nFvb@^(eKxC+&{K#6BKgR_-p*L3MA1XoUC%TQ18rrh%u zuX!}Yq1YKzU1gwq6q7D8vt;7!e0q?r$FJO+Nn7*vCv$cTu@${zaxzbQ@2R!wY61>I z(Z=w)?X6pE=fLNkZ0yhM*|@+bcJS{g_{YY^@s8uKZ?>$g8kKZ+1NDM zu3x$Qz?*$`ZC5P9z(;&mBXh7L!Lezf;(Nkj8OuuEuT_?z{EoO|zwSKcM;3qe$L~z; z>#>2n2joO{#WnE0;f=Ny<1`RGWr_3W?DuBpJRia?lY_rCXmH(A<{oK?R`zH7n^#{- z#-ckq{gY=NdwF!G6{3rEqpa6fC8Z*-?)rm`or7y9uc+qJjUS+QEUScOBc=Cm`F{N` z-yP(A>U(nQ5Y|t!kZoM=pq_Rfp&K9e*W=kgQ~$58IiUKQY|4;l`t|>s1{eum#q-~* zeoEl~mjLaweyTS9s->Tws+gEq<9I<{-W8JsR9(EyY+kMsmzcP?6`?Qxei(sB4A08S zN=qIP)g1m~&)AUF3f1V-*pSVNSNtzeE1P$PhJ{^aBu3XV<>e7&BM=HBGj@3tFGv|7D3A z|LDRwQ8uRsH~lw$_m{!_H}F5<{AW4;?T-Ik$A9kezu56_c=lg>{4YNK7a#wNkN?HT z|Kj8S9Uoh}CiyzVW$$wgF&R{=miUWpot=kLlLJ<7nt5K!l2=s~im?dr@yXQMr*z7E z^+yc)y9WfC4EZxi`j<_F1bBL^4F^Z7(IruGnv#KqzlSNb$^LMQ_4uX&>|Gi2ibc z;2))~d-LW6Cy@(&j9oWx->zx(kEsn-kqOP7AV$OrMQkhIafR8K`vJ0&V0F3w+sd;K z^&4Lk8(cX&c?COHd51ptV#M3_iKKF$@#xd5g!yUC($QG|@p{xJu6OO;A3w`6xeiO4 z9zoGcQ7TO3ta>in_m$t}3+wM2Ff`7u&K^t1bEfz9kH%e$sVy;4KMF%E@U(dh&nWm{ zc{aU}#G#yhyeEhpJLWXaAT`(VXw0v9&o&`xf6V2U+`_Fz)bLrT z$Zt2WF!e9&-}L5gj+g;8UJ)jx&tj)W`dy-*?kr=^ZISuosY3;LZCG@L>-CT5197&W|=gh5s)_}~??bhMcKAw>G6 zUb=p5qSv6(QXWeeA&8~8)N<5(m`kWC*A$bU%)GCjq%)k0G&!fMvF1(()}n*;=r+st zU7D6-8*o6Dk3K&c{@jAOWDya>qwcMvo9;8@zDgg9)UDGqFw^V9`cntm1_qAF_%!H@ zPJWCC$Azn8(Ti8Fx<$0Z>Gg2Q`VVeQr^125@OggtUB)sDMTNEX_nY`kZiDNhkbZcH z`9zYRxUkRI>mx^wJm=(5mEET__A1AJg^^~ku(t3~WbI52e1*|8k5R9n<7H%q>hqJ5 zNu7QH!pckFeb*=`BogVzOldQMBp%XbDRr`0LKd6g;dBPOC1wt%QHOkQH~az8cE>gc()gdHn5ChQkmDbKNEPVAH&UoEL&)_JF9+yxuU_G@MhA6M1ZBy9z;zM z)mB)47%t;CvIkENS6%r2{gt!Wjo`;a^l=4+;s?PtMsf$yZ0_FP-i>m)fkXG$R!K|- zX?}=FgGVgQD6Dbffr|rknp-I+`I9=#KO~e5>j%W%PDZT}TaORx^jrY1Q01N3Z9Z>B;L7h&_x z^z2GNlf{iO85Kr1bJb%01AqDFg;oTOL61;Bh6{?&EwzZW5`45qWt182V;0+MXkXoE z4$W5sWv~>}o0nR%cXUP6?d5TP&c{P^9yV&ZugBJ>YSnp1M`NWyH(GwkExHd#*(CUs z(#(0zKe=x}DOSOeOge@sL?93nOjS4wZY&o!Ntf0j>5a&@@P1x*=m%p1k3*6*< zxL4Upx3iwX{YLv#X20gN_`p&e}n7$QABQR2TFSbarE_(hzDTsmUlx;DkOZS{gJHH zh4g80cm!;Udj+^(GxYc7>~!hgx*N-J@{-VNPTS`MEN%Nylze1uU%o>V zo5IjXrS_5%`iKbQp%U~WscC}sTNu76CT0yYELTlMRMB?8=XT&Tuzrp@r~6+&bc&9= zt(+z_fjAy-g`iFl%B}?$uP7zHcq0!Xm*jp@g?D2H)5G#0b4ei|$Wie9QhP3~K{H=F zeAcm4-{Z*8X{47rk}Zl_AR_+avV0sXdoa^-7cS&AFfk7_#xJpR^o!kDqsF0{3R+ro z<4uNF9$lMostwc*i=H9E)XlT4o1R@Yut;|+a>G|Nb9~Y8ilSLeQcbTLt+_Dx-PG78 z_p~L{Vp?oa#dPHuHG}9hh+#B)J#CO$;kMnVt3fu^t)D)$$R&F(am3Yd>Nt9w@LS-cE*69tHnt zRB)k5o`g9)1ZPaZ&Cxenj6$PU1w?Jn#504;c4^ATw$N;ItTp4<9iy=vqGstj=#*PA z+^=#oNj@0ZC0C39N(pvJ2ffhzX;+kg7IrcJ`~QG`hqF) z8x*#m&hHoHQ1OaX{MbE(qjmNEy!o#>YF??k#_kch$Ip#GJ{|XgOAoZHapzhOzkP4# z(eUJ~&UVFwXURVB?f0T8Wwg*29lFK_OeB>!FGBimDbSnmgGf88t;ws-8t8#)vSu-=@bD8Du3kQRSDL-dwhn z)*gEONzUrLg*?Q<**u5f5P>GT^bS|>6fe>H2t6Rn!{`@9kv>Tnv8JN{#V+nf9~6LInGu5)?a&~3vMjKidnaWMQi_Xy%_sZWDt0Qt=`4}8TJBe~~UOM@-F zJ-ukjl`_P|GM@(ZaYL812k?S)tsbdPr!JatUx-$*aY)<{E=;DuWvb7;%QyF^ox;ZQ z?9u(+0oHOAOUu@#OTz`*-j%vvT6o-|=hbfgp|~$A{~}kxVa9``d;ZEk=(6-I3xAav z$ggix6E)^*_-qX0b^H{PeGH0;<3kd#JI(>{C5%8pz$jnx8EQ)gmYK|Ghkp?e)te@S zlA=!ct@L$pNTXfx_!t05;)Uvz@w1QsS} z3diLA2Kw&hn!Zdr{+Qt=YN1h$<36sdI$z@^)80a*WdtPBh4gL&m*qDG?hrOK9#cPC z> zjx8{G&jUtQ8dkoaVfs0*$%~^!@)c9$-WpSBrD>oA{t(-|*6zuXds&n^i=-z;CnM>I zvs~jjl_xO~wcEV*)`jsnJfTDyqKKJ_n_Y-PE)5dZr=QYRp1*c z%rnClm>aN_WV+?jee!>ccl7mmVsLCyuOv!dm0X!9!kBx8mm6rb5RR%(rTHtv+r}L5 z?=N?rhP-D~YtXARXvtOFK@v>`^6&b(c)UOQXZ;>z{}%Q@pOThwss9IIA{ic_x)6@=aeKXF-kj!t-TO98V!F;2K-ci^KO<9Sl2h?=?()3csD z-=2C&&CacOfO~<3uKhe9X__!-~J{2z(yAkf7kMf`w3^iCub_$wBf2sMN8ysM+yB?O}>wM zi5eO;g>4fU$|_($)PL#kshCdtZc=i^qvvgttHkM;n{YZ#L`Ij?$i>7$F;b8+d~-Q> z@Hu==lXN;3@tU-v?DNPg_?{9@nlqK#5Sn7j+%%6ORnQvSd&vIrOA?pu%*APj4isnw{2aSuxrP3NLI?H1L2@3Z zo{v9SusUMkUM|h4R4Sw#Qz2{IH$Ing@!~esU;m+%Doqc2XP^`X#|Ep4=t+S+vISGz z%`LQnHe;ciewJQBUX|aLX(rwEeS7n}Yeh<#-NpL2usR_>rLd^@J_QR=OaD#%A zx^#4Z=Sb(xa$Z~;cg}Dv2NfsYu zvT|2P^-?8~CQW%BX1sI5)1|92zIRua&|VetZdhovJ^CJ+_d?9q3CL&P{ufxW(;qghTex<>wECZ(1$cmHYdk-JD#UE*R#ls&*qnssd_ii zs+m1bQp4!DFD$yj+^4ZaIH{9>B9_dg2oL=Tn?a%irimvbRNbZv1WgXx*%O`*m@mEW zp_N%6d@5*y>W4~E48Cs|srKTSe>$fE#SR`3NDdWzX+uOoYa{0w4oO*p0m5Yturj7z zY3&52S0#WjX_oR0GrKx^;^0~k?r?n!W=LFpF}`-57(+Nf@~_RBpI4B40atQKR)k$8 zA$T;}FSEt(hJLmFHF`&tW197{(XJ%77cL%Yl2A;XZshNna?RWc=-?J(t2<;Iw{MQaLr+3x8nthPRPf z>Z)hC`h;W+U};PJZlHym<04HYit|)s3??K8RKG$TKD}0VAskH@N|rYv=abd+XI7)6 zj(&lNOt>wf(aBO4=wWY%9WHpcRc>CghdF(eU$<8uL&5e@xK@%tLc^G7Nl;sY2FLmg z(7)K2s&SV5corpLa4o>50b`E8lbRO7tPd;mMCYO3Ze=I4Y;`|`YnE`&@C>d+=Q(!Ps;!`!lLQ+D-3#DjaA7S z$-Qa}vUiKXNEW}Pc>%}J)E~usz=5DE#UvZZL}6^o1#aTc_I8fwORFVKxz+5~ek(t} zP}0j#Jrt0lVlN?q2`)ZO zCXM||jQdDT@kLGilf(%BS*zF2^z<-XEtUjMk!4M+XTK9(2R}0&&Cz*UHxoH49{=o? zO>BuPuI^UWeKS;Be-wUpYMv!x)K}fd1?S7%kQm8B^<`#ixj8u~+tX)S>T{936Q%)4 zrysmMl6 zXU{GK)jl*QJMu2*f;NWUfV)-#C*W2eE$a^Mc`QQsBr(f$}qgFX6Y?&?{ zZ!em!G#W%-#yr!YkH}dz7(I?qU!4?VA9|*~`2M?QMv1kZCqBv`8fI#%8EwJJ3(_0U zb}Q*fYay5m77QJM8*5X^414S4x8g92HAs#J?{6L*51aFow(zXF-+7Q}v&I3GlJOK&|uiH%}1^Ae~937e;|naslI zXJUBoKOMHCd^8Pg^xcSo@uVT{c>rx8w@6rD(V$t4)sGtIw0>}(f$BWJVVB(0oeWY! zH;3P_&#xyK$N_GBIVv{ChVBqZ=IW!*=h}UIEj9v6CWk@o5Oc_v&wh z3`(Iknj1XXi;dsf+=)6KS9#@p!vrnDNm3Vz!cDy{Dc5*% z*94)D=m_Y0Sl-rKuoRc%A#*`jB9>iQdxtuYOe34!{AjJslgN>~Hl9{V&l7Vsp<=l4 z)0}OHl{op4bEm66ayz^E_-%hB;4Td61+J@8sWEzTyk8`#S*j4R3Mz#XEl)Yu-ROT> zz9>HJjr?))DT>osfk}T#Zn6|-Mq@ydfTsR8%qN|gXn9fc+{bG-%GFV;oyc$uoI$>i zi==oB`b<1njR>@uw9P3YzGLO$SEjR0ybn@`en#Kpm}_DN(d*pvZ};)L95qMEBcUx{@c1|A_3kj z&?R`lOkqGe-A|9=-$K2DZp{2Wx%h!bP&*ZH%eHX+>f&1)L${BptfJihm_y>l%v7DE z=NiQ$y`T95D!h^C9RW zfj2|WM6Mcx<+~6wIes%wwXNiUv{PoA&z*b1nr|^JaV{%}>U}DXk$0qqlzn#I5k4H5 zJGL-yD=BaXKU>B<w)DKL&d zBIJd2O$cr4T74C0xgA#gKrP14;9Esz=TI~a>f?`o)zaPY31C}+R(%_-!RP)`#K)Q;C!IcM%ceRh{G2#qXdc9c9t(pvt+e?>JS3;(InJ$cP7M+K7Hp>Lk zshT$1305pp>QuhO=h)s z?49wF*P_+Ok00-wl;RyGxZWitB^?_G4K~+h>*MX7%__@hbN4JXo&l~?R*ju~auLyS zV(u3%u6Ht5Fwk8$bwL-lnt1-sGG6f8o;ETz?r;$M{8;L(z>NJRf-K=;Oj2DiNs{Y;lyT|wsXfgUzTL=F} zVxG4rj^ku4Hzq;k=s7CYZ?ZVM4S6Y$yjE+ToPQ&`QO&#$n{(Q~v4X0hQaLJYJ15Hn!V}&>5-eoBG=;>HEDc2kcDUyx^uV=ADk-LHWZT z($6FJO@)(rNFlXmndBYz;?xIhC6PNSoB00K8h;_Ca=FGDyV_Id7DqF~gmyU5f0*@+ zF4~mO*FB2Bgd-Sfvv+`O_u&UCWTM55KyXWzwip97ubv8-@##(L0#sR`k65dQ>i%vG z#e@9VUJ=Yeoxz!?nArh=iHu39n9}a_nR$Y^DBD0lfeO9R-`}5AAhCOf%HrS1v#rC% zW}86mB#|%)1D-Uism#j6)nMpMSYl@|ZP+!(a-;`8MT$Pl5l;*YV=EBDW<=Q-6|XXlQqZ zp@E4G*eG6Q_~#Wpsk-j^Ch1DT7hkrVziqrR`T8k@)~A)AOiv$c7%8&r)GWr958kH| zox&7^yZ9i|q%b07^zBI-^v<1{6U}#~cWz`F*4I*5m;Iv1wIz7@aW-n=MnWqsn{ z{)vp)FkFmh?Qe%^u&!+m-*$7fiV0n~CANX!{i(~I2g7Mq*^ge(EKQ9Olj@4bG({?6 z*aypQHs5uUHu`En`BM3-oYx&ps$(%WZo- zMt)m3MTLe4B$rA%W*T(|LA8PB@~XS1`!DH}f3eK?RT)M59=v=5_=2Urwi17J%eyAU z&1+*?z8}gv<)|f=(2U>M+{5>NCjQm+|Mz3Yzf?j6>v4UwXQSUrSFHx>Uc?VH$CB|h zX@E)KZRUv=XMCSH?*X!-EKeWXDmzeF5H{5~YLFrP+~K8~y0*haQrke+0dZmMF&Fg@ zGL-2<{ukGVF`7Hd_Smwuo&Ni+e}Aymw{&xpbA~-qls3M|Tc}=;O{}F(Ifb>yEY%4# z`F@blRl2v6{_H7Rhe_bSw@%ki$1^G3thNE!%`iA|dI-W1&qX zf#-Xz@-a}-nSD<*7?nitrq}-2?lF`W89anT9<;9}-o4F#a`VEuSP-N%NjN-&B>X&~ z+i_icM6*4kVL*Gm600AO?Q+zWrQ5we<%gaURbj}Ro8$(9h~16A*YgcU_>sNyQ7Wsr zz@?8Kv?p^}1Xp6?pklTn_VdLEFN2-4nyhG8L%Fv#>;&2#HY|LoT49bD3uyZi=VJxrFN84eQ;Yk!=Wr9Dd z)LaMmmuG7iPTIpF!e+Yp=cF@S{sd=_HoxC;aM#V7x9g}P{YNayb2XgdkNrvFFsZ!Bh(cXj~S>8?i{aq`i3Aq{K z@miMm!P`@W15W%)brOO7$w$L=UaqO3^bphLNXaSZu=ag3FXUsW*DO+Ph)ZRv>M8&Z zO<9! z2U`U9v~tgW!_tSUS;bzmte~4DlI4$p0W{`jfhiQ~<(Kdzc^{XVDdzJ00^COG0?hSa z51{a`L$IR8nOrnu5dRCKYim^GL5G1z&b^eJ0jul60e(dPm9ax%I3mhgctX4Srn9|z)n_g+z?sqdL4vWsR+l6ezF?O7?%g!Z(I}XV`|R~ zn6wC(H2LA%rPo^~9eban+A7kuT}#Wg>e$+RE7{6}R^aCfL);g7yx8~2tgWPyRSy&1;l#XLTClFNw_30dCsX0)xOTqf;JD-RFJD(Fd0A4G5;|Tx2MP7{K z5oCW(458f!P4Qc)*fZ0m7g*^BG={Ff+YQ+|@vdE}iY+u=|vy^2aVPrYFJU#RD zZN<+jgML>mX8Xkl<2-`WF6Z8;6t~vj`>bwjgdksMMd$$f*pp%uMA{2!K$*5@h36Up zmeTw$mro0eYrbQhVHMmul*{w$U)ZW>7|rk=1K=20k)v3Y<;xuW_Ujz z;C-t9u;^dVt|y5@mnXbdul4}7i74B&7N~iNSl4Pz`t2dWvJT17B=A zOYdrB%Gz7?zCEzB_AHSfdGxtp7xz`yB3Mg~&cttuOJ9=|Ay-{yyE=_?vv%y_{VXF2 zD6@Y%N|4iL1I-an3XZ0GE!L|!XwRrKUk&u(Lc!D{d8S@a^qa6Vzoltf#)BEy|B`lX z%LI61!z;MwN>=<4N}uB$nNqv_%${UP_m4}yLX;Bnkb`C$Lr@>#na zg%X~-sLLMY9|S0i646N*WTC7kEcpR-n-qnbu_uKiP=IBCAeY@beW=NBLtu8%^w0X! z5ur@rEpN}MGAd2S2L+$acAYPLxENk3$=Y|LRc#lxny=?h>{|1$KA+oKKzVxf2{qoP z@Q*#CAUz+QJyf1O^Q{YHN54L;r0@gb!L5*61I2~+m4>g~2N&jom!}igTuA^Nt0~%b z7o1p}4N*AwOfbLDc-)--<~9zMUp3iEg5H&LZ&`k;5XX!{c#L?lMcHG~E+@q2-pOrO z(JL?g`Rfuw+OZLRxw*DxJO@O-LV@E_sUJu`umxb{-W(3}_~^Pa9SBfb(0OAOs^sk_ zE-xZq#wAT08{~&Ay)>}xz4Lfq;4Ln{Ulc~>IJZndHknIIOdR(tf@eKH-BC;RwHc}? zP_#1AMVR|ZIZE6H0GT&-2^gP|#+9QRN`LDfv^lTx&iPt$0uOiXA6`rKI8L0047?-h zm@B%mLeZJ|N86A8?ZDWptS4MJ0uDiMcD;@kf#TciK(G#StySYZe?cFlHNI<`PW?liXD?CK%p$eJesx5 zweO*tYuC#`!;W5MlahDGxOO^z1jZ;hq8he&F1)b=99DdhuGK+W^PMU*9Aot9M_)W^ zHzH0f`uV#S-8B&=Wv!PrKss3>0lZER-E| zyJ8_!p$)+wsNfToea4ENcCsFu`P!Y$kU&~jc+EDi(7Kk0TQ%N^9>l*v*9ZVz5#HA7 zD3?Wl;-2N-F>B|$v^JYFxH^EPOThPrhJ>uC!56O(dXrv!Wqv5|ZqG?l%R`tm0!xCZI=c~xS2bJLHzz#UQ?2~6MtZF~4)CcP~33T1m zm;;=NEQ|{v#Uj)3fTn7}51z9E*(R>hZ8H?77x#Ilp9fLQRAI{RPjl^@JqUo??T6G4 zTWmp7E%5c|sgDUF=Kyk1;^oN@$uv;gXMzYQBpJ=3@xgS%%sC_(@+Y{2R14puMd!&_9;r z`4>gmIiR;e`ZE#y;Mr!!s0rPz`qeqlaC}gN71XWrp_oa6-HZGC2xs?QnhrTVklE7v zFjGv^@)--r?#$&szQw&1oRU>PF94F(^U_7R8bohjYaP8y#iYusPV*`Fa+~GeUfUNm zF-5?PT84nlA(c}Uxy7DAdc-5~wqt-D(`W*C@jsO!V06EaW zxv(9Z*UuIhcYp=R@QkD(A8V@t9pc^DDWBDxB)2q}%R27`e3_;cJG-6Kv52jHyo|*We0}9En330*$zeLpn;HQbu6xT z^hfEAkEouRT>Z29#r3W`+-nmeCPVP0UL9Yi&jaIkj0Bz_p+|LV?%88HT#zqwTZ#oZIg-F3V8CZmLpIAUuvltt4$2A%RYl{2RY$UN(f$wr z7tZ0{EJXHOY(I&+{G=>w8+}S1R)4K&v)(E*_7@Regt62n5nb9X0i)!xxRRI1!O1DZ zJIIoHU?*SAK;qaR)m^*UW;DsWS;1Ve+wlxY)L3NYF{9!?Poj<51nXr8FaL_>>>Mgr zKs?lnE#JbH*$D@xJ_EtvQ5+2qD3i+X)-Az>hxk-2{51{AR-RWVI=X7JfUxG%N;KHL zL~ak{7EA|@@JvIr8*PXR83HfY19}X4o9i?P0G{V@(Es6R)^LC)h&TO_8|yVd2c9{V z-mHNBt29*Pd1(k+p1G2DH$@TZqgyueDrwR#zqvdq1_+>M1#Fo_jqS!;wybB-I#mfB zvOJH=)kqqUy-Qw_A7{a69KfXWIsS*cGG;CVyKBA#EZ0$RKVi$J@4su=0{w^|OtO&q zhIu{TC)3f{b+37GHk_6U6u=>x}TkL+71LW$phthZ8Uc$CNdXfi72c;_V zGG{4{e$p2Q$jTJviN^rMyt@we%`Le*@musay^96NOdkk6b|-5I{$wkiMt0olH(7vz`2P^9_(N<@Iwq!2rY$)UN_ zA$@z*Ehc39Y@KZ$o*!|8V?+)E&dcA|1b3?m22$AwOqMGCSMb70RTiSfRdss+Tlyz2 z0{BJn<=Lce4cP}E-CnNr9;mCZ!-OlsY%GKdncwW}vr!N;*+XER{J?&F&NnM>7JtjB zxv@$^(+XeFEK{{3S|9Txpcs%1ne-i-5NYGzLFgO2InbdGUOCGuaP50YIkEBWK&us% zX1WtBr%3?;e4T1$DbF8(Nh3**A)ANmX)Kt&A8Yy(ZQY=Lw_*A{Q|p9f=Cexx{gX@- zj0@EOgf~wbXRHR5e3WAGU0|*6Qh@P_MS!GgBNnct>^DTYs%HH!0q=6k0GYrz0*Q5b zhyZR+s2DOqv46$F&z9l40eYFNoIbos_szEhY29HpfU2$Ypz>UEt-IUuqkZr|s!-p3 z^fETLUSq`Jkyui z{SSG8@%@q@TG}e>ZsD!+w*|9QH=uFQ!UQm0+@alBrmDV+=81s%G$juH1*F0Jj{@D< zi>7Ts(x9=q3@!=sI9`KH;ZQ(WM*Ji!AF_IBu2@J8d@6q(b9cQ7D&;s}P`1ATjKD?_ zS}JoCYFrdzrLrc%gS`1xnCAnfhigDvWXLCCd0V%A9L?RYlVyAb0I(EZZP`+)sf6zg z4b`n{aro&yScIh+0AGOi!WboI`jnNfPq<$Xcz!bVQw<4!a;EMv zs1x+J9Y_`KEd)VNrLf+Bw0Z5@x$4Bla3FcDy9ziv4Ij56 z;vv9S>Jte*3-1TAzqYK?KfvHBJwfDf);5jV#PW z+kF$y0wX=59Ak< zm6h9XA#>Xe1uFEQOlnSjR6pA;FmVttCw@bJuxMshbZG(m#VOqN$Bj+i%1(3m^@Mhl z4Y3=maB#gQENSbV#9~bfa}+2h9fOtWpKq)W7X`zE@BX;9X?SdOrIYR$%U7CbW6RBZ zb5k#(4ZH69EnU;i?oM6bE71EOu~4YjLvFNHSr@M0da;u{7fL3x7~YcrW38F9FqV`p ztfB;zr91+ECTYOj#6XsJX@%{v`bMRXr&r3?CMB_IY7#_&P{t!9#`sfa>~LJ*1VHt* zIRh$>f1>)G&F6vCgaq#lY>aS~ru;<{y|0V@1hw4i`1fb_c_EEFr%ZWWR5QfU#%+Qj zrL2OmDC>>Qx4?L;kU{yzy;_Mla75+}+C;dQ*P;|Es(*;xuH21P))-$%@~%GqvmC4@ z6T!mP(yY4R!sN8z#`(vazJq(8$SQXEikM$%4xW!MZ&7gR{nc#W{B$V{n;jFE%C;tw?rP>OAMqK}epbXYMXFZoW z7`8o^Tg`*P)BZ8r$9uq`^ty>@vjpMY2lN=N)0TLEvOtmJyzD(#+)vVrs7RU{R0O0Y`f*P8X_m(k1K#S=$UA$vjVp*Z zvh0-CFc}Z{8)AWiup;4W(&!cHU{7ZH{~#6^E(H?(!aqmVH!L53hr8>Xr6)K^L&#zj zK?rsO4k9@wFhu?bh1;7z7Rawy?q2Wc=KBXL_Z;#L@ctRA&)#;oU{&<-a^v(*Ylm5! zr8Vg%XUUrKO#`4rv$IPZcYvm4ABRI%MD+U!4<7Ei@7{z|kHY+#>(3}P#S$ZJmV#9x|Mr?@deKeVsUmwkC!aD!@JbcYU17%&LB|3yj_HHOzP=Z+o7!a_ly9)ckDKC+a?pn za(Jn$nW8E5Eg;J!yQ5s()2!zw$D=ti-4xE_0*jXikER(yfLdf$5*An7beX;auB;zr{cRH=DtKfdTa;LPyMrF#1 zZ0m9niby9N3)KOoQV&7##|I?OxS%i4>dgIev zeE~ti5co4U&_V>%r{+!BVUfANA+!h4?JMGI}V6beFoBM5r z!C~h4#dh8G$2Ugd zh2QNEdZOtwbd}%i!jd8Wnv08zsqfS$*Y;ZqwJJj^Xx6;~C$(Cs71VM(RAR|beB;qb zz8p5R9JY*|Q0MpgjgF785C$#5C^Mg)%}L4Uld^K^0t!fhmn>wrHgj#JBOLG$Fwg+w z0WeE}>Y&-r_;~BwS1pcK7^&RrrFeC-(^+t_ile~OacDO<2OjYKPC*q6(%L72sI0^|6m(;oum^)fU;us6-1QwrVJru~Npkrog>ILXUb&MQ)tu3sy zRHiNLV+{^HYkSu$A5Y&_%M(YNw+X@)XJ5g^x*Y$P zJXPM@J^KpP8?HF|d_neORaKQCD!`w0UB_PP$b7v>vM(B(HB50Xy}T*K*5HecMTElP z?pCG`B%w^e@OcAO_y6V8?3hl}IEb((a8aiB0DK*#ziICYMGyhO#OW zZ?d|)+ZU2RR_wCqVLYyGFvPcdl_dca{+{FPlW$JzjOtjd-b45;G@L(w{_bGKR7%1q z%gN%viCPaI3lBc+Yu82@G*ZMK=^{~eZ~?0|DFZZ z!qVDL&RKHb1VyIzUSl#61B4Z1q)Xwklyy~!C(_k5!&$8Iv(ig-^>Ejrcpmt2wd-tleKdwD^3Mwa^xF}b5A2)O zg0Z3&^jALTcWb}*xEwV~Cj~|5@q5yt%^DJm;?U3)_Ig^St-kQuu*}<4;5XavMrW<8 ztfCTFK2did3p)vJ)#cWJGVg3koXD3gmvT?8*Zad9f*-MZRQ!+q?65Hc{ZZGwX8eZR z*54s2>lAqG=u>IKucDx{np&$eI3F&@wZ^)>=;O!y80WH;7ME~czfnHdO7gp+Ig?bqY{|j=9a{(y{9*mzCM(X~$-~A8quG)nl}14jTUw=r z5g0@Sq`RaP=xMS23yIp{2tine?aTkp7vh7vWJbQVyMCQU!gIO1#9$z{{R+ugWOJ5=B4#v z9R6XH_hx2)OsslK5_7`SbEOm$Ia-%`ux1X@;|~D_)n3j%Y(vpX=w>8N%v2Q=?1A?? ze{%%@lpR=Z{mTth9c2VNb0jV7tveX?8h7lb_5s4=CFK7_D*LzDU?7k(s2G16x+V7e zU8@AcKUpyTPV#^o=FwR?z>pM$!oC%5-go4-C6=cTDWzhkcUJA(5?q{v65(e!FLB2v z8w&_=C;J=hxQ3e3nhIP9PFCi2)uoO56L&>5&fj&);Oi3_-&=ILuSvf7Jo(AXO4D_J zW7?~tv+q}VZf>rPvvciOnPE+R`3h)Hb;XosZ;sLvQzO|2ND4Xy4r@Ki$`gs!>`JT( zSm|hq+NwX=&(VLhCnNg){hU}gE5nmAFe32l7=E}ar8nrcmY7lyyT%0l)9$}R|B=0h zZr7ijT0=H3v@;;m6NBuhi$g7(o0W8(XJb$-AC&Eyo|SfM9EToaHzA2-gemF}ar(jF zWbEGf|Kv%$%vBBWlKCE%l2ZB`nm|)#biBE2C#`pL73S_Poi4?fGJw{ZB>4a1jelCl@k_ zXXS+iYwJkcsDG8eml)1^jT_^4j9A=dBAoRKAL-r`>_YHsY>d4fVu(sP!Xi}f_Oa;zIoQ3CA(ZYVDc>h>YaN}Mj5>W{cd zjZNY*OZYy#*?kOYMtqkM)?eb3rWh{Npp~K5p1IC>90dkf^+t#|Oo5?YIR-BM`37Tw zCk~zrwYj3PzT(sJP3eS&BkAK6#i>{BIzJb=$TT8a)3?GL!Lm=bm<$>m&flZE!U^lPy+$u5hW8P;g zK*ukX={7&~{wYSr3UiL+7)(dP5B*PK5j-QT9(B$simd>P?**mcqP4vgjv@I~Y zuIsPA1Y3Ij;zTR1n)Fm&Y;5eH_=?obK)Fp{k-c-5$%}3V?K>vQ;%o7dU~rR`y1FR{ zGYZv}M7|k%ENKcOX8wDcrNnuMtN#eLFW-qN94oc5(i3x$HSuYnA{MJWMdmqbMxrHh z=JY>lK;9t#dJd#1wl~>pkA3M1G`OgF2n5w;4)!qpE;C8=-f5om2 z_(EZwoL*QQt=&~u$CdG!cmB~dcF&|EPKJ6~Kqbp3KMn0I= z0Gk|7QXv?5_5SS_&z3P>UKe4hf%5Y5j&}&H2(UyfcbWFKQ#wxf6)$7cvP3OGA6pUBC_i^rR`CGck?e_ZJu_ny*L*Ri zaopN=NtjlaSe{HGD;*h%P3S6SKChMruY1*QvtJ-;-rm|kP#(9qw7HDyRg|pnwX5uZ z&=_%ornI7BV7hpWZ3Bqw#Blg!lP6-UgHLIihS(2UfOAPm&6qc1Xf4Wm(CvIb9>Z~RYWI~AdTGTq|=g`-<&#E5vNop~!UR`5QM5hzoj{an)Q((O7VWLSIGeKKfh#*~ALh3FpPk z=gz~KBpqDBXYU#w#Y>AN`Ar^Jn4WbW<3UYYaw8`5)35Cu%NKO3n^&`|RIMyJD|DT$ zjX!Qk*39#$*Od~)2TcvpNyhJ_g|05u74o_JbO#seVV|{edRkso)?!FS<%CgVbro^> zCX&QBCbHn>$MUh&r{&j^Jxh_oO`Q)L-&|Wz?|xTskj;^##9G!VN4PCDOzhnAYv5KV zr4>kYEu2?*Ng7{IAEJz1HVS>&IgCH)a}}^k!E@VU@gxV-h@ zB!f}F$IZuu%+=?Y(jOEJXxcbPX!!JJP*h&hqI$p1DhLP66f! z2C-^sYK~muGbs50Gruct*TzhsH~Vwq;LI#%Er-sBD?d$?b3I)r0Nnlt;!)>~$#*rE zSLjg+=A8f7%7B-;6_yPD2pO01g+|0OFhDw>6)l`O*}X29hjO&xCz(5s!hiUxh%VX2 z{Il>ON@g(@oi6i{LZ^%lyFyQ0=gp67LN;U9>ze1M@zR52uZ3Ts^tQ!RK=Nr}SF!dW zDJ-Vg>w;ofTgp}1rQzDz+u_1?QR*IwM3?V=eR*k_i5Jci@= z%=;CL7p7u!UOwqvM?93Oa5AQL=A5 z1Se|x`<=e0Yg+_YM%#@)N%S)LS8f=pmLthgbX8YJ$Gp?XtK9blf1*c*!uvk`LficA zTr)&Q@zm!!GVcq&JQ7ai&?RSP7HrH^Vu}ZUU5xJg1RKZfY`eIDYfK`pCMrj>wZqH> zuuKTW%zqwFl?&X+UEQjQ1m#vyZ26*H#?MX9#5s&LNB&d8Ndx0U^QOy2G`0+T!X00P zRPdZ!-CStGfypWqYRaKkzk$36nc4Xtq->9?w9<8kPHuEJc|JHqdNj|+=djyKze}X| zM1f6jgsFOxQvTY)n7*N*VfIP!uUf{y965s|jVM$56Ah@l`K~@uX2)r?#MXrJ^6FoI z#F+R}n|ep3*Ur(P8acddcyl#3cip3`O`R&P`f}@(O|p|q;cA6?lR{GA*jt*l;P79Y zQ9L*7@%p9YCZvTy2+60jMyGVGn*4jw!sEt|S85U!!@+$@ib9WPjZ5cixt_WTu8aZw zX8{I?@=Y1-tcMslwQD-p@e~sKFPvV6Qz&r>3Kh&Rc}}H$XV9l^7IT=l=D~^!Jk6X{%Ge9x!EW{&B5iaejG!_;+sc ziK7UhGtgvku7b9nGCt18f$*@>+{=jmTg$&bKYcOQ+ZGw-U&M*gGHW7z66K|zQjI67xi}W!c$F0u&4HvxqX6+Je zedxt0QLY-U*$G8qjC*c5m{_w67_#ep$elmKOKfBO5KAHFDKtE`d~Dp;tgX6oKIxuy z;r9Bj<+UU{DQsG>)U~zv;72zaQYV8<$;YS+D&2k+E`J#dsyL*Vm+bCVUcTg{X{;J+ zU3ynfVlq0~WF{lcd35EDjv=n@l&#d7e|JXnQA|P=YWQ-!SOm?+p2CcABi?-Xoj3I< z3yk&ZJ=rcUs#D^9-B}C)+L?DOKwyx@X9RhbP93_-lSVr|j68bxPI+zS;6Qew;yP<(Zo2n^EUV(tuqMskp%%>cvM?A;{K{dxMV>Fer$^6ix!Vl5%7zZpidlx` z*1&^rT*wBR6|v@+!DQtAn>@yA?i*7>|i)V0sbV{55X$YuJD8Su(N8@DvO zA@2+{u#0RsMGMxiLGehlahbl~iowbzaGo4`QpRN7IHqJk8Jssw_4;<@_aW1w{x-@= z3~@|sF<@cSx+}Zd_3*~x1YMbeOcYP?#MJa^W`FbU*{epqOZI*S!2;ehE&rwb;E}IYo zXZz_-qowU*&5K}4PmOCwo{@wRk62yFk7A)zshVn|q+hb1c_7H7#2)YqL2FUVhVU+R2tSKPnM&Xn!*_&%6*W%kLiG zcyqFT^PYOf*6=r{4g~dA?%siUOrLb1 z*<4VRYlq6bISrS32cuu1aD{8b)k3MQ?6$$fn0&pfS7= zg*mCIsk++_5gqBerV=}AO^QU4BocyVsfz)b^w`@;a-LPOJU&5)aahVF(IQWfn z>Gt}h;+ShIP){QY4lx^BrI@WopQEf;u_<>$FcNlF59rRQ*1y`4!on#EC%7Ib7vEn} z;5dHKRM=w!SE%HTZPzw(8V%4ywkcAmh=WgROneeS!N?UY926WJW@uM7=i_inK-i`z zerz}~-1ZUU*GLOKMpQzhE*Yw!04ugb1RZ;ob9ZU0_-Qm-!*4Juzq3}G zkgl{%U3F8{C6{m^Zocen)nhFEe48Ojwz~VFQ9ln9Y5Ej-Ei?!>TFy{R7Za%#&J6&6 z$M^-S^GCN!tJ6nr6r&3Vt1r2$%&NTM;dnf>tk!lXf#O+8VI1lq$MIO?bCkVn;1f+~ z#{K2h6zRyL*9Kwgm(__7Bn54CO6X`P;c?5Cd4zHtRi_p(3`&w|bekFv)02~bt}jiF zHP9~Sq9Qm~QwE-<7LMvgihH2C%5eSNr7m`ezi;_Z9&!=1UwRvU_5N%155ll`!zYi_ zKw8K_%U9dqaOsR@%o?zrS4eI!^l4BFs3{wgbkloxUl3m7nTt*O`#rNH?(1(||P8yp5R%8%b!{BEH$ zc+6lS!lChsnDA|oBu-NILD^WQ9#)))xw^eMCV?p5;w_!JdhrtO-qE$cE0sGQ;PVG# zpL9?db2*K6w>Tt^$q*sh<>ZLm`RQfNv_(iF=E3YU? zns{_z$iX11ycZ7gomx%IVufG`xzlHQdirZkBetcu(d#RyZ3b(twTy(!WFxE2w}pT8 zG37PPRs{v+dT7H~3~$lEVYaP@y-lYt0+SVR2JUo3arx(qZN4N#%VHGT@RJO&Pf-(y zD`uFE&npppRa!~lqo|N<(m(rs+3-l6*xGQQhtR;M5j~uNWc)Yp>$aet#STH^-k8#I^S6C zcC!d`EsvR)Fjf-l@xAXYg(v5=EH|vERab75x@#&^%zSd8(Kvq^dp^si2454fEj-8X zX?ov5+mU>sSJ9l2C5h|z&T;c7=Q0-Z+IGKqDbtM45!}-dl_7OZtgW~WFxd$zZ_a=W zaFZx!{_^InIv(aF?OFBmr#m1EyNWtnHx81D_NXSTICcz^Z;dMnQ>s4VX~|5?(>EG< zOR_02@w$rgN{5=c+sW!15g?k*py>YWkpJQq8gpFJ>%rNpEzoXSQW-7lf= zi5+W-6h|)97MzEr?$nQ)A?3;wm#B)H-s)>F{nU)nFMrYdK32D~;Er?lDmFpk71@hXXM4SP&472wPqf9t$mhi;kaiN$b3S`M+>blYT*SH16Eh1j*iB$YjCogjDwiZKgCZt+bGuGP&x+vYowZ@7cli1AWZxa=GI2je^Jke{A&@5XVE;9pC z`Zo4as&?WYTC*zUrx-TdE*mbTYdp+Gyg##r?N5pSG(H6Pvn?+}BO9uF)I_&e|JJa* ze;0o56t8R25%ZKE+1L|3e_MF0nqwpzulx3BQESw6_s5TQY(02(#e5O(LCF>nKljdB&B(@7Xf*0-KA3 z)#D+cFZ#m8A-5%iJT^y0tf4?$A(!U}rCjq8qUQ+`EI%B}a=AUq%;p2SKI_*Gz&&!ia1>LSvOab zDNvjEIvN(^=C_iyAmvdUov zyMl+0Ry!|P7OjB$M<%Y7Zroj)K3wRisOU1NSbk+ieM)_CP*0{a9Q_C<@yb44-1<>12V~q^uxf z7XblQbTYZQ2bD`(lxKzB#@2O9rbKpPwVTMuvNC(VZ>^$q(H5^7B1Ll({^Gh%<-(FV z6M{+vcjBJNyi@8lowfs*WS!07RX37oe9*W5Rv~DiR&T}E&WC@DGEtKkI(S1knBq}DE z+qU|}H?tQOQmr}Nx5P!2uIM|>4lMW;&9HQd3_WF#Az%=!S_i07^|FY^Nx11eGx|Au@Ckchq~iXCtY4EkU0v`Y*62ci8(d zhlZ8~B?86fcrVAWwa!6_QOH*tl5mY!O$#Yu zwD~q+3YRowaE^MNKZs1i?8*`El4*+T`FT14aFpKaD2Uc_LGJHo-k-oH(8xWr5KsQp z;#>Oiwy=wi$z^bV+gbH$r3`qc<^aRyqBy;tji{cS1JA&tI-bUp!C2Qswh)Rc^KW(} zrlyf)<$|C(;+~Xbr;i<6vXI|4AGO$}N9W>z!_q)&+aZcaVjy_2$yAf6=$`R2OG`Lx z!K-n8$sH&AIze9s+u7|<2E+(!TIEf|!wb`r8ppR2H%ER*VmpO`( zo(C+x*3_%?EN`96ONi2ErY*4ldn-wF!1$w2R?!;~i=TP$fPsN0wp;C614r14H$)dQ zHRJn9+!uAF;x#2Qgfbb@@}Ho%0}&F5nJH`){lDEZ%5ercXXYtd!z$Yq2i!6XQZMGB zI3^h^x(eMqWd=%t^l%iVY=zb2@pEX|My8mWH7+e*>cCu_ZZa#nD#fHm8S=sdL%z5( z5ZR%lZGs4vV0m}WY}`}m;izeFa1_mzQ&3qGt-*L)tb3hN>x`msK~&TMZg*Y_kH@k(^}1 zE-P-MitAIJxV8`QX^N$Cx0Pi@uEFvhfSI(&p<*ps$EHO(TPCbDySAT+I)3eVnpqu7 zwjIaMl;;t8;QRBPi+-4nj2iNF%zyxA6k*F@5k)^!Zo_eEqmoXx?q>m7M$CHa7-jv= zoojg~i=r`}_Cr2k)0v$2Ggk_bX^>YF01w zd;Iq7JfjYBluoZTkJW?3iEvNIY|xm}CERq|tgVaQQ;_HAWoOhXTBvO|JD6toQOQ+wuXF| zjlL7DO)U3LY~hA%_v+rXAWG|qTkDx2TO*#>VeqVlVh11y}Ddd*xh|NWX zSw=}U_^x&6+l9g-!#Y)~ktCNqS8Gm*5=k`pyMmMxLhL~$m>fwmCaKv}hyfQ2Edx{+ zV(HpMeBhbJz^;_qg1LJn%WdJ#E%9=R%N)pL!xTN8+j3lYcc#oF7J?FmqCqN8DPRq% z`x|XxwU@&&n-{Lp8}8(Ml<=zHmq}A*%M9+4^9Z2g^X9p#GD!Ls%f>}oFbtL8UaQDTy0$nS ziSV%48|BqY^%kd67T^{Z?*5&w(zhWQrd$FSc|c`E?*WbNmB7ihCA)@y%cvXc_oB7ilB>6iozS+ zw6ub^z&$wmy|$$diw*Pk{aW_(X@ou-~y&(HS5ZWSr(Mk$g75Hv0UFVi8J;mT53W9Q?8$PV7tlCC5nmuI zJ=xtN=Fn>DFZKYHQW0|J(W_jyJ;Z^C)V8406UN_amz5=LR@OA5#Fo2i08ZGxF!^Zt zk1rWg3~yTGz?JYJ3({5KzN;Nki5ZHwFD~d}*SuDTUPMau7gOt)_jm0Y30Y5c%jeP7 zb3ybpn+n?qWy*giB5Olb(zv;?owkC3u4=lLR>-v)H}D!*#PWRl%WB4ace3w-WSTP= z$x7nZmhvKLv{++S7Ok?xx%J!M^B%Yr&ix^;+>tCTq(ILExNYs|vY{q>BWua2jLA!| zbYnAl3qa*UsC?2Fl|=C@l_{NNlbe(C(df4iPBL23b`vG6-Oy$cHShArgGa|cgn-~# zi~$*S+2lr7EJIh>NU*+wSTsYjh8jz|R=#f2TOW4`kdjU;HbI2N&RM5)W=56E%AwsF zeqoQchP;R}TImMaWv^4$8B-!U{Jzy@&E#M|u%~~&QyZ?DL>QG3+pSMsGy}g*^9QFH=GYKz4UJIPZ|N8vPKvf4e%k5h{ z8nMW(k5^1120VB|=KEaFqpdpG@k^sDNnC^%OcCwSdbsPft<2<>*S1 zwm(dDe2^;iX#Q{01oo^%hUP^gEJGUG`>>=u#Gz>HT*~Rm+j=pwQj0;0Q`DEA}cgy$n@!935-hrl^hgNoxv5`g+q*k4oPN>6^@{Gb0lrTje6#z#8Ia~*q zg@M<|hLZCIqpCYh6o>pCS5f?k=&KDqc?*UvpctpT>`VVJ5VbRj5m0+XW}Tj1m>Z=( zW>vp(Yq*y$>GR8oV+n7B7ZCw2x4E5oXFWxFzJ(+`CRV$F-0^xOlRGiSrFJpu1MJf~ zvYWawnVF43Ha+y*`DRT~jXQQv5r1Z-+I*^v079Kd$ZmacB2hc`90_Mzpla6gIg*M8 z^&e^G=#RXcZhY=_Mk~{ol%dL+rzi;-JD(Svhi`6)_3HUNiJZP+zJ9(glBKg$v`jy^f>^aB zk72LQb@%+NQqj=NH!3gxAc;=NJ#fl29RLgDZG=??>C+X>excN_qn%p?_(rp^(s$30 zF67A}@mg9l3N7@E86@5a^$L`tMJ-$reHpcAz5y;xq%A1ot62RKdPa&A!7p~B>r3~) zdNq<5-EwtjcIB-=F8Nb(?_2DDsrOu?bFW|ypit`jwvDRcZ&#>x{AUp#WID0v0@Y++ zrV{oNN1K0ptRLtH$b@AF9fdO>fV;%Aroqw97jt+_uV_KNfS6i>#jL zHX)Vw-hlmr7?Sh|^S!XAQgTuw`9)M5&Tu`)=g*L5C@-meB&+@cu5|2#pg#4J39jz| zFI$*dk{t2x{9-3}l$6>@J&)pI1U z4+0ayyAJRzG(hO}fQw&9AzTaHY}*qx5GGepw>)%j$lu!^yB0iQq`iSxFy&HKx41I_ z_w(!0ZuG7x8Hp+}g)K^GD5N<8w>qi#Pte8@)oUBJ=DUnk{A^&!I_yrRVI0 z9fn!MV2~CAx&n%V3(39?7qXF@7^@F8_4V~Fxc7?|;0^D;(C+gqXh~L$i#0em@+}AH z&n9NQxjJWmM=CX#8xAgDrkmWAa*5K7iiaM1aF~~oNQDPt z(I8>Hwc@o?*vh$7(DwR|py0M#mucs{z2f)3ePGEx1h2VyrH%x?(b*Dnih}9A-u(j% zBJ=At2&TwlTU34Z>Q(EZFVB_uu4lYYZx0|;q@mw73&KAwcriXm>h9X^HgLFdTN+=0 z{-eTvL23~4tK2|2Epk-({{XloPhZduwU3HW}jKIe<|I`VPk>f1dc z{^A4dHGcZV$-5{5WCeO9K!VO>W)$=s-pSY5>Yu4*+}m}$56oyM@w-Q2(Ee9&Tu{*) zaNkyj$8aoDH{aQN`9orWT3KJ8%+mMttXMz+tUsqvW05(1zMz> z39!ePt>0^~o4{kNI?_QojB_mCaiZm1IjaEW0a7$lx>po;2x$whQ-poN&WInwlbzj( z`0iEG%hmH%SBogd#$>8YK8f-1Vq*= zCfeZtihKhdrPn@4B-gB|;EL(@ye{^m`+UG2Pz%Rs2zG_&W8H}ZeMa?ep~CD=u@DwX zB?H;Had4-JEnoSfTC-#A2M%X&55z~5gE&BK72N$sG1k@Oxy;XU?O}|dfWSvaQ8l~Y zpfDlwBT*^Ra$&5guctmJxD9_D?)1bsbrTTKf27$jRFVgU zgdlJ@Ol76-DTxZ*UYWPYho&|_Pe@HB6;#*j?aDEYG>iq#t6c8FKTHFjP!DD@G77?w z)NSDxn67O0Pga}l@wcoP`r~2b3#t>eT8~#}hZ?uT!k8fqnR|U=V#2vY*Gy!C=^r@3 zKKJ?*VLw2$+DQO8>P&B6slljk6~}D-qkMb`QUM+V3g~{h@Q9)JEU@;RheRs4=rn zAR?1hkNuY-fL%LF%>M_?PK4&GNXb`3FuOBu)>v5%}i#c4IEy`y>M!LIG$7X@b{$V00G% z8;o$94^L|x`yJ`r^Q$`o;bPlmG7EG>ba;OmQR|@6!36`pz$xoe-7}i%6mrE5kFGA~ z5SERPg@^|52YJ3HMOx<^YP9#qVZ6Q$c)e@c8CD?t7PwiALBKHI zGROnmBwdXI*TU8VoKbj;0p)29*`7F=W0SpwS^Xe!Q0=}kmFv$60NB+w|M@5^oU(TF z&8c5;&)NYS91 z-@kWnb$*{1K%_Y-KLl?8S;LG~NPXPl4~i!5!5{Pv%}ZUFM#xEF+X#S)GSf0Nv;{6g z1Q2SRgdn+l3Al}D3ZzyGo;-nLJZ%*AEFmZ@Sv$YQ+a*XS;!}|ZQi0`_ncq7%`$_NV zXWbD!vZ~UxM@RGrhM!Y@!N*ud+x$SZmmLC$G7eNF^Hv7jnm~i=SuSpF$K5n$()|a{ z@=i4!fsqH;`s>rGjmgxEl8Rs23|d|*_JsI_gO@slM!I1m1>^_u!1G-jhO7hJQPZ#! z+Br{ta?f=oB34s?bc?Wcm$9Pawq1gHedy>ONELdh`8=GmbM;WnA@jfu}3+BM!Y)e7r}nUsr26ZLt%*Y>c%7tX2-S-@ zyud98&H%yF16(C*;NLHMfvan@bag}Lg<-Hg;etpe%BjL^yXr=hR_W3fxUgr?L6!mZ z6F992&~N(+4`f5Z-{E6y)?*P4e&tIY`qL%@_=?2tW_$qwHft~r{H@mhXc3$1;nsM9 zA}x0X4}zkB22O6_&K(;^Y3b)sjmLQqP1*z8MoS2^m|<8hDmZL`M+rm197)}!*KZtc zlerX>-4t-Au2FUYXNvzwYFQ_UNdYovY9c9Gh{tCUl(l=njTZNHb+=!2SFZk{glvH| z(2KjB2jFh2yC9R7;^}2Bfu}NfhY3mEy4(2Gxbxq(kCK`w@4H3;ccBv~Nd$ca+5C;y zY78F_k?4bza6d>~EQ4!Gnt;34$KBD@Wu8^Bo*L=7J|T4PCpEA|3)a<7NUHxHlxi4J zE{VzLPM>h|p#yy+gAjtf&22*5W4#MB07_l#Dt^$wTnZpoP#!irFK>scyBoBO^n-|? z0Nmo;4J-r>T=^Zz!H36?gTn$j-()-M&nunlV>>5XC3@F~w~|aHIX9Y}@kd<6)oeqg zVt4-V(Ua!Dd{^WiZvh0iDeQanMlnp_SNJ#atwr@x0Kc;V{^c^xj6h*%ak^Dw4_{pJ z$}w*~lcg06H!5?!J?iDDvh%`Y21*#2n0mi48CgSEM7~NeF1CDGkL%JUvIiFC=Ktg3 zq+H6*PG7sTc3#>wMOHE#&p~J{lB@a+zmJ^2g610lLdpeQUbO(|CX$w(KINnLP!9hk zfLZ^j8xKJ)+xFTCp;&*fcfVKDMe+&e{4O80Fi|tGf@b0>Bx2EC+pfKz@5}2P$+)CU z_zkdrnj~5t`eu`yn{ly;+3Rkp;z#pX_Uw5NdvnHaXA=bGZEbY`fgS*-7G!I7ltt@_ zZG*P;&%fhjUNqDw9oUqbq_Lz3u%tuct+)36`DGNc{!k3Uc`6$;EBzuzRjmGbqot}! zQy}>K`EzHGk-H|@1i>&`TL0RXv}7+mIOvlbaAarqNo& z02ov=U;*UgA3li!1pmxI#>~s{Q^fQ;)x~iUs}63=ADN-~++znE6b;%g!&dWX+tejN z*FOO#>y8pj9*9ncpcTQzAS^f^7F#l-r&R+vrawd_K^IXc=PUt*3dJktyxEs8Zo13-J&FlZjuYU@VxGeIDYHU3bSM%e5kxSKgmfs zh>5Je+|{1Yf2cVDeB0dFxq#Jm_?ygf;|JMq(`w%jZSj#RWSFm@|%(F6&_q-_$f&EyXm`1kDhT%5~6S{QFPHEVpCT~VPez|#Qz643DM z{6lK|-hjrNrn|(mK#JS|FhifJ0~WW37YWP5X$|ka1$u)00o*1#KGs)dWuvK$;S%-B`Qs>W&#d?S6tKg97wt1ihsC&*!l_Za1Y@*-g70 zYkcJ)U33Odt$;zWX}ZnBYWPeBc-1gna=KOCeHKgiBfV}9Ge;iGPnI5gj`yIEF-PE= zhuaUI`+UGfrj|#4L$_mC!Zx+<{pToSGQj%1oYDv7_U^ge@VZG(y7)?oXb8c@c`pz8 zNgwqq>-}r#fOs2h&sR~g;IXgQ>c&rTU7>7t{%ZYOO9YZn(*Qmco4&(vfacloQ+Db3 zxP0FV(?HD=Kg@Mmd)QCXr4v$GLi{zdfZkuWHJi+Ls)m10i1f-N^tPL);Je~ zw9w7Z>L`EIV$inIj;X4X(W78s9ovu=Y1w;Z)SkYcWCK*#8ut{ zeydydbG(9bya_&5rtKOjVR;z(66LeZ?aV(HNg0s`d|5}(`EEY8c@1w4AqPW=Qb0(s z`;XxEn#7Bh#yrXAp^?1d@I5I9E&>xEJXRTo+i#^DcSb7i zt}zerU+llN(zJZQA4JBO0>(v*e%C!9MmMF6rS6w!^ytlSS=3h-1zHv*-BOJ^TD$N-Y)=5~x`eSB=$(cteE`>PBARpg^?a_Yd=4Y-@z z+U~vSTxCM`6q_X;!M*GLrk(g0udLYf!6S(sTZ#GmtJt5J?H<+ZRl}XyX~fRhKXORM z2d3wO00YODL<$@xa2UHP@$I@TU_NJPeJU^3;^LzA779Q@`fL1wswVzjK!4HfDqYIl zR%`2>sbLFt9)gPb1MzM|A6Y6P;ARgHg|oP{_14-=x*v(UF3@zfrPX}{AN_$1jF*Ak zG_t6{4}30;mJGL*5(rwS3(}~r?$ay}3|lbojrd8B72s;L5i|PK@1*!eZ&00wF$)?~ z5PZk$VC!PX-4ED+m^2bgZ1~jEB_JIu$3L1r@7}Z=XkY%|zbAVs994F@DXtg`LBeK` z!sKOznLU;)n@oJbik^D5Z$)2&_2XX3sqWL9Ebl^gOnRy=R~_~k7* zyiWo}#WxwvEiFGwTY)(24eFkreN+0+i;w6D2NF{Q9ziJlcy(}bW#ctrr)5RWxS9C( zcNrz-(S%?Zl#BpnK1gyLO#cQGGX@Kv+BQ`C(5dFUyidiOlHj-W)Esca+d%n>jQi5U zF)zNKw{z_eeeQk4>56Av5OQ7u#`NjKh=VSa8*xR7Wj0@7+nQ3hi+Se#_@uQ1Sr&F` z1E36kFU1SsNdV007m8>4TaiOECMo-Uh&MICRy0I7s19sp7I<=7HJJ7}bur`t$Jc-h z?83@}gLVI>L}?(Cx6;|Zdd~S+{Z{wc7e+rkF72BQ1NJJrI|kmj-yNXgW0Cf6#6Cy{Y2C>4(o|k!a|Gx0?r9`UfU)+@e&ofTK*E?Y z*R&%Nf4RmP8x+)XL4e>uy@Z5KN6vwjKne5*`qe+-dcApZ)t$~hl^fUgg`q?PkRxPX z4WL%>3OE(g{Jj3}fz3)~{JCNUl z!3{0~;!Rp}><=tp|3*>(^2cMCwl(_Y6Jb`N6aO`!j*(#BnYjt~_V-N>9M%;+&vQq$ zyp-o&JQ>}i6tW(8p5G%PDEIRv@vp1CJldIcS=#uYD4H4p_#!8F^fvzcU~rCeD!tA3 zK(E(a{^Kcc>&c+-p#wQIZ)7K=IrWTg?;9*xRdm9nRXRxdbfW%Czq@C^(Ojs%LrmZi zV3}`ddueaTzn)j~MK2S!*bg~~y>>(lgkz?!|`0x6^PKaHF z`RY#6x`)ykSQy+T>cSosk!9d$d^-_R<@>sVRtaj|9x#}U-f?iOw4RI%fA`Sl(--`U zk)I)dA{Zg}5BRnFT#3P8%@NR6JaYIg`pV`OUQL9F%#m~b74SkZkC$1?K{Ej zyr5PDG@8TJ7V9QS5qn*2PJ#i8he9!plHR<14?9MlxS(C2D=FO4!QX>-n!&^{m!aI`dfAlg@P+ed--?rN#gzKsISk>J<<2yH?IlUnu+x_M~EvgaNn z@^&mjL4A*2?C-$37v}rprtxP5946jo>hbmBdl_EaP5tW+bTMUbZL?Soh|J})uv&G< zn&Gw8MPl(j6;lKjMDDs<<#aI-_}c6LVW5E1avKWS+kp@#b?_st@T#o>(kClhpPcc_ zf=HG(`=PG{tcQ=U-x&;`8Ou*Z;s1!lLM#>OrYL`aN4T?5R*@OqhBI72?HG@f`!?bK zl6wvG!?a3ejz-0{QE*cfo*o?M!^7RJDM z@xBBIu0Fzht(b9tee)}7m5)bQiG==-ip>INKfN@D#eWJ6PR^NqEqhhfG_Z0$eS3Pv zm{3}LNeV=G?jck1ew_^Yh5Xr3FYQA*FpeSHvkZ7-EImjagu#VF@lp!;8rm_Zo0lAZ z^;jBx0^>h)cQ&ie#q8&b&_%TDFlOz-r?q>>2p&22r0R23Uo=Dh_S{VRb`+V5pD14H zIf2wGek)ZXWCVeSaK37GMQP6dDxMRl)xmbJjEvsh|FmE1Dte=*wg)Z>_dDalgQyha zj^O`@Yys+0u|)s&(|zGBQ8R8>MsXc0E0asBOiD;_b(uS0yp#Z2fB)qhgU_!39(;9r z*Y%Q4q8}6N{Xy`(VCYpq5av@WXuShp_wrx)RclZfTTVK5P8OF!P=Em_AnWQgQLT6J ztN=qFrO@aQ>iK*`up)PbEyAiebW;x}MabY&gV4uQ{Y_lBq~`wq|L7afg;`pe&v&XW zc)dUH@7jP*HV0TN_q{1f0>&ZkK$Z2hJnk z7sB7DDgjyENTa9^TADz5kIzbAPK_PqWZpgk= z3x(|cDlEK&d;@zk4S0rC_3z33#XnH5TJOS6{k@<}|A(R#0ouLqeGdq4KHn#=ho5W$ zxZ?h9q%}0*ZB1TOL7M|DVi0vEA?`PSGICJ4#sIO?I2e1@JeRg zXBA;`Xpl@+MaGK3?i`Z;3E0{H4^7^G4rI$KSKp`W+EP-1zmHO`z{g{Cji% zJMS^-P+#2?JKH?x@P`^6^UJ--f@SaL=aK-w43}l)?WbX4=m-%z`Crlz9ZAMY@r?sn zRa&GsW{`I9f~xfX8@edKp_)4RZd-j`|4)uo6{wgCwUM#-NEis;6*?WAQKKybrl>m~ zm~tl_PVX->4mo^L=MT);Ifq87s|WJj`D+BTNQw#-%9D9Oo9B0dpoCI@OD?c9EyA-u zZ0aTZ?Aoa4MnR9nc4Rx|tTzGQLjBSKkjCEgX)+t0Sl&o$65!c(w#y&iX6p^FYR9V> z^d-nEJ99yr;Omf`tJrnNSH4UMu=GkG=2&h~5`Ik&W#VFmqaN(XJpxQ!MR5aQ=* zBQ2yD{|q23@%3lGx6)|r`YkuF$V$?2wnGl-m&1)h<9;d-EHa2ZR&97khwtpCgxT4k z01rCPups*18?6U~G~h81Qo5NaRnz@rP}W^(v8x%$|G`fH;{zS$-1G^-1Bdy!iV4%!EvyMFD5Q4%ukwE*97#V2RPyw@ ziU}oTpiId>_$+Q~^n>G6c6nJs;1gDmvFou%G!k}4ElZ1TUX!5w++yA59dXWrwSf>D zsObY5T4KEP2nP^fRyE3cB=Pj@K;BIAojO_Nh3P-LvpRO2Hslk?mN&D=z&Qh1G zzPruCQTs+|=)=AeX~qHam2NSr5cDcYzwZIeum?bs)Urnbul}+>WkIsftE9x1EMZey zPJ$KuE~f%)#2&2*cE9*S@UH`^;F}LZs)YSh_qotfX>iM_U5k$F2@05^a_0S^Y_vuY zr(mS?_C3#LkV_ky;Y^HwIW)sMJWNPC0M{F+{t=r`YIv;&gXd7|=xE_e>jS<}^`>Ya z#KC}bBh>=`d#}n zzp&72aQNcxX#p_{!L;=xrXDtuI=(&qcRw*M3$DG2k+Wy9Acp5Kfl*y)4N+pf#k2QA z41!gNnn@@byLQp+P8d*UC-?roJva0k>=${5>>10pc=5-f8GCz&WMVuysild0s$^&a zWAQ=^aPi43fu+du)QuPXP#f+Jku&KY06g~K zKj1My2EHol)vuK=@}L5dEDF8;TCiZLTtdidY}ubJdkRg*T{>gi;Fkk=2V@L7MyhsA zo_Z$Owc?5|5-UmbpV!@zkN4s!j(h|;=zbfrT-3-x418&62^3!*oH%hpRWKzjEjK;g zFN>nDt!=oCD>dl$yDyyLu5CgM5|}XTmXv1uCbr#9XKhC@Ph8#0wEUt`*qJi|V8oZ` znc0?kijCGkb{Q2pBe}xU=4|vfBEm8UC<3Mno5%KXU_)u)MqXR_4UvdWi^=*>{*0oc zE-NI z%O@t*5A|YnRiO__LwhxhnZO{38 zb+Oo$nQ*GFB*w&4foTu-p@9hyc7cXTITehB0;r@fpiO}lw1MZoeofKqIu>a&{PUfV z6IKHO+LiQe3a2pu{{TL1IWz-|fdhC?Q}F+VI4~3-ag3dzW!e8!0T_dF`H^}!vvr+B z)=^?vD>%A&$1frz4umi7;SLgM*`@I6rj8aPiLM8_=+mMzou8lYZX1Lljgcb2uZsI|(Yi=n;}fJvD`Wcotm>Xg0K)i2JD ziWPy;maXzcf+S-f#@f`Cmmb)Qa}BP5p*A97jC>{IXG%dO9v9~?PBCg_{kGX-6jO|! zbYio$zN9p}keZf>eLpz%?b$$a0HsuNe|M!7MnN-?7q4 zQGI=*X-BwpP+&5AH`ICi|JeJ=uq@ZDTSQP0QIM2Y6eIA1o&5T(>Nh+gm*j_XZH|{A6Qb38Eo{?Zu$D2Lk0$ z-oY1zcH(VJi@k&Y&|r`A6wALm*K)wP6}9;QYk5v>GNv zp4GO0`gE86RoZBD(A)W}@l+RI^Ba4hg#+1| z;lqs#SU$aTxO=I>hhG~4C>J({Et{XSe*AGMbtZW&y>AM2mddnzsW{3`Sm@yAz5V^M z<7Es-&&>?|0hmHvps;IJZEr`U1>^&VDk=n0M+oR-bF19<`DU1MGs_SesWt|69oy%w z>Fe41Puh5Acxttm6m678j)7>y0|OgBdk&A0ajgw$RlMS&{qN->=vJ1xw}v9g(uY(h zZDefnYa508vhBVALHR4o&(HZwy&d1SxEL3VsI3Z`FV6OD1^?DI1f&cS^}Q{{k1E89 zc($Kk;mdom(9<`*=K=AOnD4XaSQ+tB1=DUS?bv{Nm#= zY-eDxF>2pw1fJfFrx7cj$NTatvY1#Q+b|Pjtqhu>qnwL6Qulg#%2jGrxj|{KdBAhT zp)bO_^&B3It&P<3N=)s-`{EtoJUH^Z-S$|)VKh*1OCxpWgLesxrxYNw2oa$*ts8yS zN1UX;BsqATW`Solg^|}otA0lXQ<+FZt%I9PYCW1!lIo_}{*2wAZb*8Leo5;>GcyZQ z^~lT<=)`FJTv6$}s~qnJT-RmeyH&>)n*ok}<$y19VT#Kpot=J1cLmSt)S-@ZcW;(< z@K&>|4ganmXqIj4S4X}5K6a~wrGggkCi?rtyAYk^Xc-b?5CY^uwWCQh9onGgoi>Sh zUkpM46~%U3kFu~%kIfI?+#=7ZF&LzXr1DdLKsT~h%4N~%8V-;2RTkvn)ZtfBx~Q^A z*xGI*7@str;axyxoP+s{Pymruqa_;qZgh^S3?<(BavnbJIipwF_!8_5GitY=C;}}1 zLY}uCyh`oT*Ebzi-*l57*f;ka#4sw~QI(^Qr^!RlmL7Brb{Ts?x7jNIdMM8v)NSNj zo70oLw!qwD%B}ZfqzMOy8I?MwZXo|M4?S82jcFlz{GLQ6(^2O+t9#nTQDo*>Y5AEf znupG03z;s>=VVrjGCYN~A&a$qQRhlR(9y-Q*acCl#=+RPUjX*K zPt-vXKS~ug`l+Z!K}za%fy^C6%+BGY_&9VsEka2_$-3)@=<)A+rQeSY|7%(pM3ILG z4^(J25gAYKxN~HqkLz%Ci4RmLInR&Cy`*^ZYCmtF+@*|>C(R7)?hH0=>%s0!9KWDO z(J?TdZHV2rrqyk9xaS1886MBm$tmCsHs5QsUx{^FlbT6R!yQg?78^O3-yKludi)f( zRlUnlmEWo-xiZR2C678I9vS+|M8ZXg7f#Z}2N^&@qc| z?78b#@B~pBhuXxgp42ewv~r|;3EuY&P;=0T<({3vJyS-X_VNcgDrAClL;GGAf8@8J z+@c>Bh8)+;sARJYsH`>0qEc_K>^*N-+T^pJ?cmx}v})cYl(Y|a87K}d%ne!`sYza2 zs4|_cPGL}dtYkMM2$CahGTnqX9;X@1K6ptv&FuH1@&&nMk1!qC#At6J5bBDJ{lgUm zorb!WUp*IH9e)w@QhO>jusGv$bZxNZPx+XOXG#_l9*Wbv#<=^EwSH&}hw^ZDk$kUU*>@83m)aUVei-6WHO!Ej;DjlC zm!=4F!&5hF|2v&rkD4Oh?B`33udn>Hb+h}RWsY+rmxRIj_(`Yvq_t9e8_vy&jtJq6 z&A!t(pAPtU(vCkf@}zzr=N_>SE-0&TmFE~$jnhAvAgQJL7b5Vm61H!uQ^#`ej+oPj z9mwH^M`AqsZc1CvkCd0}=hZuvTzG$M)_D-4Ah7x`GyEi&!8+5fesoP8x0kX z-Eo@3_?r$ED1cfHTymbx;nqm`-fyimzB=MG=D@|tti|-wB4j%u;p*7Wbm4>V7nd1g zJk%AwXj^}*k0?;7`$BY}{AR6ud*LL6Ber&VAIj3IiaDOPa=Rh~G#Cyc}y3~RhRBNI#25#}(Go9tc z{W#6qv1*@Pv+hYu55btm|3D#uRkVR7_mf<5Gg#dKaw&ZB)Lns*A7?~3lcd+>8PiHq z(chfNd#7%<#k`MwHTquNnwQdD82HNGI9zkd0 z4(1Qc@Gf&n>$BF@`znxnKkhK6D3-uu`clzOY<06O*Z9wUErB+q&apyKuM7=U2{Tr! z{_8UB+LHIzEQP}^^E(g!s;}{0F(EjN1OJEf=*RdPRc^bUTz7@iw!b5Kj3zlR%q^;L zuiJE}WW>6@Y~a6qbl3<3blv?&-la>%{D9rfwCvTWqqXm{h3n!$whvPAd%Qe^cy)EpO=*FC;` z`SRGi%lfc#{iRDz&il2Xx6`CO4s8PKEe14PN=KrCP-a;lUZ@&46ZiA&6tO}<&AfP3 z%EPlRuP^Yd<#7F+m0%WJ7goNra|@XWV7U1PU!!F-cfpLvQXE^R#bd(a9RJpZk5f4x&;u5z|rj!QA5mT^&DR z8p{<@rV-_`AG*HOaxr{84BEaJ2r2u1bsNY!QZTPW@^iss%VTFo_}C2OuAdjkbfewa zA!Ptr+=dOMx0U5L?$7d)?Hz$20Z)04g*{N?Ynipah%`C`4)fl9c20eOg4@FdJ&qq|uCKX{wsl*5k@75{IxhN_f6;X`|Dt=t znlVdpLibqSr_ST4h40Z{mXe_R)lq~V?7OzuyK=FxQGvp+gj%S>SpAO7b94N1^7Nfp+$ zy5}ya zWCjB`ASNqcFkFau4t~}Z416;(Oq9e|M2eyUDL+HOGSo_5JAG2(gN7=s~G@(cEmE+G4u<{91ai(-Mk&^xsgb%+CBxFUjZhzC0D1g_4J;6 zz1|I@{NW)4wKb+@TG6dUmYrz9+FzNj8Ifp(rSB;TL(yH8fz$mcMQbzC_WOyhtx`T%=_E*Gi^6I2t(q_g4VipkW58_dE;U)pln@zH+j z?3373QgS;Gw3q2z;$NeDTysiud&e}HRCj)z_%;4R^4rF;y(|Y%V6~T7*%Y*L&4F61 zi{3X3;l>|AWBa4qU9euGuRbQZ-40{Eb7{i#i?2m~zzAIdvj%DKnR?oWNk1=JDg!^C ze@W#PWR=Ln!`3eYXrN$KnH!`Kxcf3?@_{JX|cR%(>|9Ntgx@s?HyN4~A+u}&WB zmoK2R15gIe!V%J$wC)B#DCt>vZk`{xD%92S8pLf>7df9#Em`} zlpud!{2kGzU-lY&i@cAm-t|9BX1soxQ^uR19)yHy>I#~yGM++Ty7Vf0u5*c3N%lzr zDCL+m;|ut)-%0d!EELrN!4hREisoA!JNg(43Hy$PT7JEW>}o#>yL;O#$x(>nH#1H; zOJT=dGnl=!RO}k>C#jQBk?oyWD`z;rqX>RnZ zq!f1^zeYnH5xgR|232V5q8c$`&1?I^Y1X-5Vu*qBdE3<%E8((n`1`)<1-hO1b2N&# zP&+K|wmp2u$&2{-gP;upb@?!v6X{NQt~V+_61=a!As zfaL%_v$kC;r)vwYlgXdK1=R-*nRgE#6Cp+~S=&t!>ueUnJO)75Z@uyL zo|i~9_75Y6Ep61f|KVhZeC!4dAls({!r)FkQQ`0P{^K7M%G5CbTzc#HKV0N~*~tyE z#$%ceeWiROvW?Epzw??tHvCaI%wU(b&;IE|^xMUwQXrH@8NPij;)(%CS_rYOediWM z%l*I2>HhrK>v$cQRTw0iM8BI)iv00O?^FS_&s5i|{BOS#f#3b}cfYxM{rSxPc`pB~ zm)|xbvhV-dNB_i^KSS!TV2<_fpE&VPocJeB{IehaIUoMXBmck5uMH9|eY<~Bb{M?E zlD+lynpuDD15K+J7oEWATSu*jR0MtXDAFt?BHe_;{M(25d`tjGBn%Q>(@E1x1Y{CV z?`KLsKtZ|ndCZSi-0w@1?{>7Pm&oxs`0x6a{DZ^ZYKghAt``ofO2@=$9RF5F{}Fo6 zpFh=Va;CxIG4WqM%D??kh@8<#Y5DJmH$*K111v1B3w{gZ%>PkR!6%rafQ*5oNF?=d z5grM5H8ps6Xq2RVwVHngzu$i7KtFY=gE*cMX!8GY#VFJj8Z}Li4QFg#5#A8AU z$|9nA?LYtU6#D6VB53voI%iGp{Ff^d!GU5?+bdo2KfVz@>dXr-wAq@{3vJ&2?TU!u zDHNlr2L1Nb{<;}>xTiW$wv^GB|Luxc;3-(0O27JF);=C7Dcs8GNj@H&|8_-MNEwZ; zBKEs|^RK&NW5b{$BTfB&@>b%R(RFoo=Rh~$3l*XYwFkCb-Oxx557l#-E37?3o&=`b z*4_+*wtm<`QkZzA?`r@#iiNI2#Re!{rl{*VU-u{DWNZaxPczUJk!fZwkR=1(qiU){ z=u69i%#Ekov~4407=lH3u;@R)IFlg4KWc(>sSyxPSpZS8Gy`QCGwAfo>hE=h_t44&# zw9!d3u?#$KJmrr!^^Zd;?Y~|ROOm3GDZF3oF~XI(_P(np=Q`sb_Q>|MY@m_25gd1L zZ6CkyM)Sj9xlk}w@hTiN&){_Vm}a%aBd$z;l>l`CTL&~28NA=bL3^|AwuWszQ8&19 zy~7kEHzp1AM_?#EXc89)=9EtOaP+7s`%5r5USWHRF}NyJKAYg%1rMj2r3vgZy8x$; z6QqljEDyDQU%_?*b8vNAXZVfqk2m-7$#MMOTqP~*6emvxZ5{ghd{KiB*oBkjnLz^p zU%uG_AL9+^!J6FE2bjeQ+biHh#rKP5_IHaBOeD9&Q z(?41Z>(;W!3P}N6lhA2L_;90p1ALfLz%`R6tNLi~um0qaM%H>UL_sHbJt%B*fW z7NPY7lz*^}Kmw80RXqLt&dBRs9Nn_PMT!|VSBOVOxrZzZm=y|dl*8-JpE5q~3oznv zWnIIc{g;S+SnIPmTlz}U1Hbi%hi|3wds*=nU)#eQFm88090TdemxXP73rV2p$9QjIK8wqK{YuSqVyzGNp;<+M zX2p>635kF-7+|LwYi?lNqpB{~`KjQ(Px-2|kFfmyTL?~7(Sa2hgDiNSiqj0$5pQy4 z+Sv<5M^(AXHpQHFs!{+WPu#1^%h_*IFv!IzoOCB9$QSilX(BBsDJL9Db#?*0lQpq+ zj59^ZGW1pJ4N7=qa-OAAmvPkXDST~NvoSyF4t+|w)}YEA}Ghob_7{ol%0t%iKS|s zW2tH%0}2jyX3Plo`SU2p1Ho+wDkhU|ZMT}48CHloKWqVqP1?b3!{9B}c}$d}dl*qx zLh4<=kK`;BlO+aD-1ENsg@(=vYJ};sCufHu;ZtbHWn_R0??Z==9#@t>H;P_#8g(|= z!rX;GpOlswTPCDkx9Lcc_#I_+jSUS0G25QUeRe%mBcbs&;~6F+iuS4$Q!y=JsR1|? zM*9qZ(6h}V(T|+p(bBJ@~r039=c3hXat9RvcR95x;R_AEz z$5MTn^Cffkzla#H!Fnmr9Q_XDKe^6z(?PRTIq(?MKWw(rCPA|zFFFkoFTMhOmvV#j zS#F7l)anJ-LQ^v?w7~$QciZREcST6PX#3SWH&BNOCQrC7!uFhJ$mqkuL)M`?al!qS z{ASkYiClVq?AS&Vi2~SdKQ6_dEf+~Wj@^g0gwlBie51aZ$xfU6uXJ>m$rq1vqdirT z)!Fk*l^Ee*s?2!58aJ`=OeNl4T6qK(z{ILl$ocI@`x8Q%@zC1x(&WP%mZY5KDh@1V zH#nnUff~1q3860$e>pyEv~_ijBN{V8k6)@%3g8aMJktcB;!;D)PiOV-q(alsW7;!F zaae|K&F+3?$x43z`yKb*uViT3S!6P54CX8e1vm z+P22GgpzH|RY5Kx&)W_AYaRD4w$>yJxhmXyUv#gBG=53Z@jP7SJB@L$`OcOP?~=7> zV@Qq8Pggzg&bGG9?SNRolPQOB#!qCmvTF$1Nen3;Vx4TO=f!a_us8^BRO$0=3b{_a zrtbpLuSnL)jjlzC$;LV0`-S_P^#4qP#qkP*Z+BaVf}d`;`B_{o@yNJQe)#MyW_(gc z^rZtgBh!NS@!uQ1SF;(aR;AxM#!iwkAzOJIvR?;KfINt9(pH=P_o1wo*V!un#tX zMrA+V%5;HPRxIT$Ef4YLNYH%Z=EEGqI?soi*iSDV))m`ZU-^j3J9H@|iVAVMg_niyhMf4cL+0XmTT8W5IrRLAuHbn#lYAd;Q7c@(TcaDOw&J&^Jj|92F zRwMQ44+dz$It|;-!C5Ke)B7G#Am>{u>|P5KBp@w>y~XMZI*Hv#|Ksu!SRrAq&C%h^T-@=2Nmc((UeCaK-`lMm%5?%DCW8BUgO4TATyvCey-)Q$zW;d_&Tk-_^+%k=&8E_58`4i+u@i+Yc1F$>U4ANpjk zqQ{RsaenXyiBilfeAI#@9^2((n~Vem`%`9+H&|$*eY+<4F(2mcpv;L7$&Cjmym<~fnt2Ucx}b0)|M zkWjt^38xCKhc!60X_D+rZ0od}%&Ve5Q-=*vkqWEmFp7OtAK&t|M53D6=97BOT-P#` zesvt|nUm19s`olHMNp6~#=eX3KKt0u?^!s?#o0g1Rhf}mL}UQ77rL~JwLoUq*3H&0 z1zWf-Ao+$`A)XWq`pdSJl#R2i7!{K7&(K#-Th$Y(x3rBz^II0l%E@VSK^vdN z=HUjA3c-4B*MHX(m#d6n9QRTwm^=-Eh4>LN6gpPzlPR?VuFkI18{Ff?SA_D9)qDWPCPll$9_lFTHBB{vG5chTE zQzLc!S&GLZNLrFFZ3DTKS%|9I<+BDES;bA>Nw)O1NZkWO24}h0BOY-%lp5Cgt0>a2 zMNmpsU~hKCP(|!UIN-cBSt*)jd*+)h} zUZ>#@LQ4=2pwM(1xP`hx+d;ijuyet8rQ+!kyc!GZ9YJ`A3|d~|KGr@UmuT*@(lSZ~ z+I3HSYJ!jiZv(0{LHDqFxm}r?l^ltX42I)94%IYH^Pq3oo`uXMLLLTWI2;%IGwWx8 z-xKQ|cL7u1fd*un@{Rw6_zzk>$@@Znuk!O}qb&I>;ky|Y@ovceAtglbZ2b<$O;g!e z!^N^$GJFC`vG{A=x1Et{LK#D(kskKZq@CW@m5vLjr^5B_OOT}t){n2V7i$U_>Lf#M zbz5wETXOa6G*IXJQl%OOZI1%(AW6m^q=9Fq85$N?0)WJC0}dYvz9}FHU)?EFj8AHh zo^|BP+HiRUNDp1SGtt4DH+-bDCh*6?@PGQL4LbY8UP4IjZKK@=`$-+~F0!+iEiT96~ucBb+GH1$B0K0N?xvY;(ch0HgD@GKtdnL=7N1%Rs#* z-TQ<&N2w44vT&$dT={^0-VLJ+k=2V4Xegm`g=_9xh!^gv2(p|PAr`Ad z_ur_x>08?lih?{Kg==O8rq&dZU(8j_QEDDxxGMb*LWEycn_gg3Z0~SJs>PZ&1h2Bi zfc|LV!|H0ADp>a5bLL8?Zb`Kge?X`HOgaU679J?zp6!Aui^!rQzh(E%DVu>uVRBi~ zx&=g=igYLx^SC?Lo16eU@^dGT>Bak3p?!!clneN>b`dJe28jn!#Y{u<-LflD-00Co zOmznuMZz1+P7!FPtLfUvzWsN|!$1;y7F{@rHMQw4eB`$u6bbRrSj;v4mp=rcDXlr} z7E<>8`zI4(Vu<*Xh5lC%1r&zj@6A6+hY8&O{8KT&W)faJ`Y*r1c<~MppEqUEjQ9TY zHoQdc119$2!ATAFZ`Xsmavk8Q|BG%B_1cH&e?FS{yqY1K>KLTV<^b>JLLhl@wmq~^ zrOEtc39^m~fVN~?m;CXN2U&n`QC?388gG9-C6TX~y#O3=z=8HKkSV*=u)}l_IA&&W zoJqcfvLT*#MlEjEWg$zwG8ATk99q9y!(r26;m_GCN$tLIztuwR`Xmx$>cTPzBN$g{J{I;0{`Nhs*U8E z$)H6l-BBvu0P}VjScTf~Zbl{oeq*uJ#9!Khs-=s{bulPr9U^F4DDa=|I2^_B4L}CP zvs8V2H1Hh=abGq7U~!yRp$J26R~jL{ta+lPL-zXD-M6Dx?~kl>5hw@*J?h#kFv-|c zUssFA7d@)I=%IL=?{M;zI*4Im6|?bU{rWTb`)BvZlf(3+!3 zIkt74vxiZS)&S2*f9@1GE;Ec~CTft!qNoR-v!>_DN&okfk0(LF2;$W#Bkk#tao2(R zW(V}K-8QF_9ceWJaaO7DOvwNGggLHv^z`(oR2kp$x6wqRJCao9-)n)C#VlBk%WV2F zr*p%0Kj-uFYawYVk&IRl05w?>#FijD-u`0oV*}Y_2D#X>9R2eByT5^t9lR(A4g$J> z`^*6fC693axPn~^XQKe>DmY1r5I^+P-RYcCx|M>C!&kVW2#Lb)LzB7!v`h2++&^(Y91t}H7s0%ir(LSVjI zUb`s4=Fz&@-9UE)7KEJPHvcq))YxJ`{495=k9Q>az`42`2EpR4vXPJavX+_deO2&y zdsUx(1m+_UD71-!)>FrKDokG}3&1`zpp2I`Zi^fmTXZ53rhNuK&3BE(C_;TXnxN)~F*QZIsa#mDp^vJic2497}vSjKH`<`&xg zrmSm`RiqX;B|2c6DgF57Sk`yD>m5U!kPxUh&$D=XQo4dAMHZD^lvVMZ&w4+6T*lD6 z6=Q8nHsq*68TYQ64i^CwXe*R^CDKHr5@U1u{HQcbsLHJ{_dvE!H`dp9^^ z(szAu*yT;G?}R~aP&Pje?}l|K^-^J$S>@$Mtw$lME{i_yuy2wPgqq0$N_iO><~-4& z;l1ZaLXLVKQzd$XP{S!*uZgfBo8etM*l84&XRlTKb_|1tl=KR=b=dULZ-B}fN5tL3 z{L-dh6sIdzNhgq{WcU^$eJ=PYql!379zK0)-bKgh^+^mUl!54?+ZFKlsbGMIBTBLX zRt*n1UoB6DghtK5;!=Ql;ioRh`3e(|p^PlN7G>!YiU=2XA+HzMg5rqLg)+wF#AfGr z0B){4(jBLLrL|~&2(WnS=?gsZ51fHV=;UKP7-U${hLtPEy@&NsP0Pescfe=}J|$}E zwe@~TXh>R@G^cjXu|B&9$qZgWgj3kwe*dm5V7t{xkdg4>oN4?h-nWyBJ%i;k%}|!1 zu2q#nDn!!-YbxRsYr~MA^&)sy8AMtY53DWDK^iLyRy>9dg8Ep&J!;x4V!8YZA)ys% zUHP}0k70C;UyKBS`~_WifiElx^Ae)5oMLRnsVZiwTJF0L)UuI!1__odB;hs0NshmT zErVA$D(GfC+07x;H*`aKQ6s&jT~~80kRoV0kac7;tshgUDGFv?%NbGadzJ$(*Lenu z8pteEntfTvKpeiE$C-&mF~gY=RcEW$&aHx8q#G=IONw&^p#OK(D(eRT;}%7xb{XbR z7wv@{2{`8Axb>f>7OGf)aFQkbuvqW1M1e~{5I2=DG!Mw9D-mjK{+b(%(@ya0G8)*{*WoE!BmP1VK8ZZ0M-1^K3 zm`7qa)VzwVV=Z@O%|lXW`8;9pc_stuf22~Bfpb~ADFf72ex)|g^bNqyfx&n=pb@~N zF18zLnU8EkS4SIO`7IB!tC=b=l-ZCxb7!Y@VXr5RuD3thogq_pPx*XKJjXygvypF&7)R$f1wDxGJ#H6n|j(VT}j zGVD89el$T+{lU4?Bf0Aqr>fSgUYliG!56%*Bw`WcW4F%Re+1aH8c`M&T@cayD?L2# zf`S8=uRbw?a8gm+D{p}CREKdFAd;=~Nn*&+$CWYz!{3ohWGwh50Jy(9yLpw3#OSSB zS#W9-^VB3Ebue>%VmaYHv9h1@pzi#N5lP@}X#FHZy<1tDX6*d!!+p*L_5o!D2D+>! zW%qHe29j3g!fXDo(_CaA>}uL*HaH7cihw0oxX1nJA@@%HVOaJ<8gho5e?& zeqRS~H1iByMWaN`R3YuZ>`1|j8tU8{e{SAOJJs4=EPP&uzh%^W}>ht7)?El&g9pWS;4wdp|EBB!(OsScNgNIwUrppu)YmWjvc z?gKf?xoUr`&N$d~O9Wvpf4}G^2E36bqg(OFzQ%nPhwtY5+VO(mMn@rVyE**7DXVW7 zUa4Om%B$oFeHd#4Xqdu9!41Vh80hV8x!oRZlPx6S#o=9o40ab_qYSUciJP^AvvWov zks)$ZoaxYE(Qts^QU`Nrcgqvm>36ABh4X&ad+t(8nacWYGJ42+1hfsxjQzcI%0G@l z2!beVVjeiA3%+`Tc7JzfCYE7+k|@4N1ytkRNcQM(GjL$yA~Yj~9DF@s;5Le{+HLrY zz&cFw1mElxkjUr=|G?0I?QhEh-Kw92w=hHc=Vt4fCm4C_2sG8_ml$c2;~5}gc4)b_={(;z1{KUa9g9c?Au$qC;_DG! z!5KAyzu#{yGoF$8wAouJy06AA?S15ep0J7M8pv%dH3l1 zn`YYM3~1bj%NhFU$9Hp_=Wi31*I1m=$}s}1PG|vM2Oa>U=G5P*IM4>5#q_9%nEkb4 z9;fN%^G|m%>S1Uh=2K`aN#fKr*uV7}rOaB#on@eXyxSdjR7iIWu|r$88o-f;xh_5T zRZkMbtyMRH?tY|?-1G1?>zY1zuEK@QUm6{2%%&=A#Tnb{EirGf76bsm#okZE*so|X zj4Y1_20A?+P;t*`RRc?L;Xm4Goo~`*XC5EtQ*)H1!t;>gCMb*wpG!*3kH({-(1qX z%i&$W?Ok8}gM>0(Eq|_gA6@44=_s4*=^XtV(+fF9ebkAR3t;vrjbM!FFUyebGdy!` zf*Wo6eo}7@c~DgtWkzFA4h!DAdRrej02vD>@7qg+36swJ(f6j`&xm{CZ>Sdq3CjA9g6 z)gRYCjm75~2I#IJUP}OW^5PsMzAa6qYfVfUcS@nY5Q=(KRqDwM=hK=QG6sYUiEMR2 z(%zyu4y}RML592von3kN8P9M54WQRjwq@dn)UHq# zkE5Prc^tG68tq}m$)t}tz#S+UNl-5{T0v3Hpj9TmFqF#!aU-AjBYU>FY%Jo?!sU=& zk2VWluu){rT&izmPeN^cAZ!44jNp?+_eXgdfnn&ljb$KiQ3)r*jf_25Vqp{a;|#af zO-~`RD8?gcMt2itxu~rYDg%k&#WF&Q>XPFcNO};O9IC}AmhlBf$hgIvRS-)A4y(sqfC#)3%=>x9m_>%Q=ZO zC=VhS`_Shl690Q?&SHPo@b6s9No8&Byvv6E1{q2sUKD_Kh?32zyf=}?k!D?-eLBIt zpf2;`@%;v*R&|JDZBx+$d%y^jHm!I-r?V@Infx!*(bE%J()qT~<7&p_fe0U<0_7S>rOx(FvG?BqzMn;j}nj0$Y;C2r0lg$5>9 z;=ZE|1A3Wr$sv)tPTpWvb?V{XLcbtoqHazlBp{kGi3@v3m4hpX7kp1Fm(M%gUCMK( zj`{>{S;_~q;$%=%D7snREOYXl=_G-L<-M~A+Uvh=rACqXm0AvD`3h?Fq-PyoUk@d1 z>ao%sxr#=GS|?Td4j_$W=Q;I1MunCx!CU(uC74%DJi)6~+4XPTi)+qQ?UUy||HR(q zdCyLjIX+V=UVM_(T3{BUS!Lyb=HwqoA#!N{K##I6Gdt-c0z4E8hcepuqY?4b+IG6{4{4FZ)`KMc!_4Z>VPJbejAzC*TAhtQBx zzt-RtS_2n!?KneVl5cAhp{rSUe65Iy;CS}xL7ytJUsL*r3 zLVTzGdkOT{{n#tPZ3OQH{Cx=j{n7uW;QRYY@G-z`{PP;Dup+6L7X*IR{Ly=Z7FKe1YPU_JKIHuAn`Hnj z>}Tm<{wkthMAnGj%ep)NH&Pvc<0n+7C;8vd2B5XX`efgl5x}=d57CB+&-M-eb<7na z=S4`_-_?(4L=6I#Ac#Wd0i<*+R?yIA;Scq0W_b)9l4O99Y3}udj!+k5QhL^J-u4`A zbSnVH{s2J^v#Y;pq7d9@z~mT^qv)zte;fO)hZHers5`XKE;VHhA-Mi)mEJf9bY6M} zsvr=a({}rM3(|BRXaRO3XnhEfZ?vI0&RDLw$!whqg$*Ti(bO)XQKE${01Zk8x{|Vk zP`YD^oUPe>d=BG`uNS+>+2)WZv45XwzfM}1Mb{L;=@1GAE8oBN2Wl06c79$uuEIrs z&$8uqTE@aU$Lcq`R4HRUMqF~P+@VLAAEzTvs{VAH>E)FbHvule)rM3v*(v9AsTZj^ zvcgi4#$EW6>k73(C*KKXUFh*vQ;uxE(gf6dRy=}BA-s~kvwPyX>mz67Wo*Vahd8z> zJ$oKr78>v~n}4(=uYB{P1R0m%L}cHYrY0aLgihS=-DqczOX*&r3|@AbOZ=uPLaL^z zXIw1ReL+S(W;)W?K(GAMoE9>ufx3i)2!CeV z`pM*1zmDt3AbQe3I?i+Nj?3nnRoBaQ-R_@-!pA?0HhNq4Q!mDTyxLl^ifQ&ashgw4 z#K;80+WmXo#*p@yY@8p<3!!SBu>L38)k4zShx>_zE7IM`sxuTi!T0J(KRGv#%Fl}X z&Ke7I^V@m^0ILo4{u&z$Ch#wF4=AR;FZ=5VO`<*+KdYI0^F}dH_3Z&YN`<>;qJ+LY zuET_cb4ww4Z8<`2abIyAlL3!Pv$~2r_B97}HQQ7~+tF;uszeSpRkd%FMJOnz4cc#X zN~Ep;RgH7V=z75VOW}hD(M$EXp!FuXSAWyv{)kGZD~362krvA}KP(-`Fl5BS9%(uw z^xct1Qutnoktskz8>!aopK;i=-DTqmYDY$uTFk4z(nUQ)O)syB|J)M>;Gr8KxnWAs zj){@zau{gesCu$n(sforZ&=sCyjWo`nX`*IN?&Na?DO%|O;0I=h~k)imfG#H585HH zjUzGBj3$u*i)kpjBFjEERqZcFReBnJEEiaR`)SZRPXt8Vf|H&nMlBe}098~yzeO%8XU zi?$0zl@D=`dB8$WF1@^o-1i(X)SHFQs_8oPAXNE>+&XJ+^)scXhy!s8DNsK2cttL}dU;TJZ=l^H6f|gyIDlwn64yu|r!tg1kI)$RKjBKwaH53YC~P_euHP(O&8}PZ%sst z`;p!+U}?%Ie6fVE*vy-n)DaXSTe`zA+MSU_vz+l%h;6ZibBLF9m22IMbVs}el%v+A zHhCU-I@nk3RlQkzDnigq>u&K^t{$f5EV@k3O6_kvo%vqC=RP<|QGMxYj+VOj_P_#y z;y4UJqmQFaC@AOY$W_lo73^Y@T}IT3_)RCvAs_*d77Y;{FKZoVl0F$OdjOfHo7GX&7hA9pPq4CufGX%PazD{tfq*=Ux66Ov)}NdsWm9u#%+=uH;{bl^RBvgB<{9<`{);f zI0wMa_0+#5K3$Z?E6jM3NH;|6@Dye%$U+IoSRU8-p5P1JTSH1+LOLgv4E zuWleGGem0=BR=uwbW?my>{}0aeUE+BtV3!<9F*-Z9wTEvfIocXXyxcmY6_gswAe=& zeJm0UrVom0&~cK0NYU7Hm_VdUL91ks=?QpS!BpJ^e&z)}YNp#QZ1;+@I*+q~KH2nx ze|2crH#E2N9G8~9D*9>7k|R?hB9`Qv1MoN%^?P`_=uzEeQ*5u?h9xN~$)q z%zZmeDT!F>4PDRQ#G8X58p#ke%e`st>WCFJ_kiqD;7DpY;;#AY?+^BM1@JG2Ulzm3 zn7!Z$2G$R=suM6Z*{2*Btu5-;8gqt&g`V!MrTBWRF~y`MVwa>5O@*~Ky{P5pIbPdZ*3qL#i3BlGs7bt_4MLi-bA zo;Hmxn}_XPw{6@9wTmPBj(MjG?7uD9q-dsCE}KkW3o%aAzIjVk?sAsK>(H4_m)uL? zvcd9vX1xiOmoQqi=){@YNVRBRM7~n1Tpjf%?(y#eo!2m2>PssnrgBs)WK6Xkiv>(t zR~t->3xxuSKOszjqF1K7NbX}C$I#iu9hxX+RCpnAY)yMnCqurZX8;QxTN4pefR$U7H{J*OvaGDgBU7R81n}Tcs z499&MUs;9F*!Z;9Nv<9$e>@e&cxPbIF6Yug%lwG$h|eQUz<)$4y;y&4Ui|~hgiPz6 zVthZYKst#q&7_2#UzldxgoK5?K645d#u2<@2J5EKL?*L?@T@}O=3b_(;RftAQIfq` zW3u+31ROW}IzQ@>O$Lt}67ig6A>uylQi=SJ5~(XXRfWp@Bz#0H?YnK>%`Z?nPn`9@ zV4GJy*a=(}USD3t&wS-2i6OT}c*;woL_7-3 zv<@MbcMe_TMJEpFHR-YHY~}M#z3;(Zzq?*6>psGhqil*b(>S%QYCG7`-+89Z8w|f}bS5>tNlCcaCa=hkarlFnc*4 z)heX7J5j*k{Sv>z@9WozGZKe0dc%^h+hcYFd` zmCk36^o_$^#P?EEw9I=6SC5azj%8L4FKzm-_+$A@ zGYAb$yiiZ>ao=(#ubl-!L{?8;*8qferoEY>bN=iT;C%P~$h!+!)G`Ly81X!7$5#yW@PcxeNhQ~%<>dyC=v%-l*_hkjX-PpOjUmtix z8pwKlpE8Ths8U$@;Da^cfuE76OrjxE3K1#{U$qYWQic@M)yoX%^Fdch zsj|j=80gx42vHx@&IR2mfGDc|VQ-!Sti z`W}Wl4Cw{Y3%Mrt-AQuuF0o8rt0oQP@Vt09D2J!2bcaAE9y{L)75r#ao2yL}9d&k!Sr4T@{3< zefK%b-Jm1174CrW1e^tpv)YyIJ2U!C5^5Vw74*+ICzM;UKc>(% zb(O|veodU9G=u4b$_1o8&E;;VInPDMhFn}{G=Ymg;ti&lVL2qtX~c%_^)c3nz40@B z$%RLfn|3PA68*WF^a8dg5eA@quj%3CIxqXM1P8-WtwWe)`e4jdyb}<*(MeP-1e_); zw+7m>nHdHxxoO=`iPue?7_=TX! zyxqAeyuyh$(i^5_p0=l(u~6^1fFIq~4GfvLEDdka1u^4&x#QQ|1pxt$zYIv3@w*?t z)cVd}L4^l#mxVfoC2*>+*ZY;j6x;cl&clN61SZ$jMvv8(?(F)9PyB4dQup!**Ow;O zDNgx8CszEC=ys1`lXgx+afO8kvxHOo?C7YhPekClN`wf5eoJiPb(PqJBCFft&Q61F z!tF8muNLB}^?gotxbc=XNifYFL{6V-=NU333cvVW@nE|%BJ%u5;&j%zL{}#v`#VlW z{@H8J>7H>7Tc$l46c=dQql6`xM#0BAkI2N2B&sCT&&zoD$7>PuLbUATmX_wGI3u?o zlK2`i&^alnvBWST7hG#^1@2qvVC1cv^qv9PGb72Sx)CK;E62TY8oq`G9$!AyR#5~q zHf7Us>+G@iiFVB;V%z&tbP?B>_-*9Kn08Gp+|F!q1lUzj-BCia9e1Me4>|l_{^H=m z7+ivwp}tbya})h7-->#gy9c=*$aJIb4Od1ct&S-Mtq_=TetEc-SQ1f2a=Og+LZ{MA z1!uPFy+#L(#>yB1YlIH2&{|dUdq9@fzjqs*qBRNYlj!!|9&t(QBL5XEker7J6RlcT8-SnoXxi_KaOg@B-xaQ*XVfuM=8JAUi~0rt}=1u5B0o~=nGLKIPsWs4yk#UB?ngX);> z^~UCRwwS&6T0QGCGGzNAx#H9q~6Dbm~m8_b9lJ&-=H(pD2yr<#pH| z{($59!@u2sS;_A{#l6R$NfYp+{n6Zo0?aTL*W0UC)EZWqQw}JMv}KbB1C5nD0zx8I zGA1Xv)-O{w)EjZxF*`#9ph24-`vsQ7KSXz@v|xkzFCkgyh2(y2Z+!Os)t#i-wgKua zkH}ryk?!58Q(j+`&zo-*DpaUZ+dqZMKI1kG$?5o346`*Ou~EmfA}?{NgI?gq2?#&e zi%ZpAPMxcK*)FRaSJh`R%3Sv<$ZU}4f@~eV^$}?+(QuhgXEHZ>Jie*#YrlyxF{)X= zh$>n!aP5)wTOiV%q)is7w~H`C+l~v9HBM(Sv1SJ5Va*>J67JDf&kQ z?DehjlAV=;M=YNwy-r_{WEYjli*ma1SxB3uMMf8C=yOI%p@aIW}y2Gk?Tvrj&^Hp14ZkD}a zyV32?mcDSges3e0Q%&fFzK2k~h;+-7xYRunv#QEYyYr9OI=Wvm7|Y*wUuq8t*R~Vp zH`bZ?%(2R0cX7kKCtAO7xLFxPC{a~gmY9IyqL;YLrvn$Ug$`3pQi4uVvM(cJ)#abR znW4Y}_Q8J09GyHr>u!DKjAs3rn&XMXcO|m{2BtO)bh^#TiBkWky)zAGGu^{DPIOGG zOG@pf=!~juQq)o*9cw#EF*TN2%81l@C`F{UE|x?nwbYWN(uL7bger<^MeRaMY3+Mx zG!#`!&hz%1I{nb=d_7+#pK@K<-n@CAC(m>LfA{^zB|N_{vBn}V9g}!3GdK=`Ef2(G zG!}CKB-?M7#aujQy-^((psxn}@m?isiRogd8u~wEUlDE^cqr2Q;#Zj0Qe7pnwibk1a57)i zQ>bA1aRoLe7nV)dMOX0@7y4a-DV=DrHj&x`1@~^gjjtcq0!pcJ_=aQxdzvE-AIBW; zNhuG_sa0)izwIBfls)^!2RoJJUFvRvGN`k%$=Nufc7husJ;!V*FV@N$LkdYV_tUk5 zP|gt?oguA8S?h%2FH3Suph+TUL;qdKg|?G1Ft>&TJ>$88nVmV7Q8)c%%sUYrCXm;% za~A>MKK!vj^lBROC{26UCD`7@qUEsUJGW!-9>-JbQp0}EzC_c&#+r6l@7>h7@}cd( zY#E~S?yKdnEX-d|x+j)=l1)?=pK9PHUghdF{-}GIUV-Pq(0M>?gId+X^mybHe^cG2 zK$Hnd&sA9HaJeoe4Y*y&3b~lj0spjFB~X&Khzy^)`ti*i`7S~x>5NU1)Xk}4`W`25 zc5{!MdZN_um}9NW)>hR*?Rsf|Jf3{2@oF8ws%HVF{PMUl2{+#g)>eK)U`8G62!zxe z51O_+d91Q%m+~W&h#5f^1$L2@Za4b_VGv}psw^P!j4{?F+qjhIx9I3UxG7e{ae$3d zCD#DkZ7^i0Wb}9y94{Wo=XH8t%m85~zr8l$T^Cqk9JXZo>NBbmC3%2rK=J{pWEAYs zESN1C-Ng~hg>;YrfNs42P}KZz7!EIF?IFIk8VTi#*hU0oLrx?O`L6RWx6r3kV!zI_ zQK(#$^h|s^v!&p1FLI3yC&aO*I4>Q4pDYeu)ZwE5C$**>NsR=Fs4TB5t&iMpBa{LI zy6>?pf_K4>y!Tz1sdCb-)y0h|cE9?KosQmQ9Jok4D1GJjf*pXGSo5vboWy63~FKM zhO6|E0z2}jJCT5nS;cxW7d5L=rY?ds+z!#QX@1f3kE*0;x=Xg0*NMADEVM_j?QryE z-9phe18g$fh?1zIvXDen~UA6)Tz_(k$gFf*?I? zr=3MLUsYE8A=`C|4=Bo6(Z};FHwll_X`=s z&8?Q{U*Lm}5K2(;kax2GumN7JHnjH)@@LTj&;+EkZ>-ALyNS)`1J5LELN0YWupObD z+TG8ORK^E@l_J{1=!2S?+S{{h!+SwLFIG9xmV)q`->LL4Ik8-vhNL#mK$^Bk63TVc za|l3vsgT!tei;(w`s;E*(d99A>!txiAdxaB?wNBlPg*Ezkq6`40n*ho5}mS(`wW_P zM?q?A{|9NNE>gd*;ke(P7f%?|=!i=SH)6HHcASMbQ^*anZc+Gs=YvciMJ)V9<`cwy z_WQ@fq}|k%VIx*i_NF%fTd4Mv0D7=S?%3WWs~m{)yAvkbf#E%H@`YO~Y3!N%!zXTi zOo8JQ1{?S_H=UUHums!F@6D`|4EG}G*nDljppr!PW}eqxc#vUk&0WD&&|h3HxX@}nY#JfR;Dg2>B**~72wGG z3!5+269ACX!lEXzTqrlGyxO~FpOcpShkYuaEJPHqnjO*|nwHgvn8#!%WD||BfpUs{$qAC@HA%xWyf;fXJcpT@_xVj2oaDbuw zPzdKpMFSh%m(>ohCs4fJ<;SgJU>4c8M^vi**))wJ#7Ye|P(9pn(y9Pg)pTm`-eU_S zPZ9!dqc>J|)FGAOc_D_^V9pCye0V!+{rc9?F2>l zMW>k|90T=OHgpN7_fy&UB=1?3ALr5@3Sl6qn;}n*uL&p`dTfGHmA-O@DN(N7Y~`|= zbjX#S+~#;{C7648BB_%qruIY|9q2-oT(``(aUm+G@$yYVLE=(<_9|a5jDrW_c!p(T zrBe8n=R8lUk?dfiRD1}%i$zoN3~GJ{YEby<$G$p@GGK9=D`mg8dd#-oNKW!wVA7Bd zQ7gCde)@wDDUnJC;srsV_6sHIQL@e8QY-+0r@OCkd#uwA#HmstdqeM2(OMRWH{?{+HW3;L5 U8j|*Uf(P7A8Cw_?{puR?FBm*SR{#J2 diff --git a/chapters/12-outcome-model.qmd b/chapters/11-outcome-model.qmd similarity index 100% rename from chapters/12-outcome-model.qmd rename to chapters/11-outcome-model.qmd diff --git a/chapters/13-other-exposures.qmd b/chapters/12-other-exposures.qmd similarity index 100% rename from chapters/13-other-exposures.qmd rename to chapters/12-other-exposures.qmd diff --git a/chapters/15-g-comp.qmd b/chapters/13-g-comp.qmd similarity index 100% rename from chapters/15-g-comp.qmd rename to chapters/13-g-comp.qmd diff --git a/chapters/14-categorical-exposures.qmd b/chapters/14-categorical-exposures.qmd deleted file mode 100644 index e69de29..0000000 diff --git a/chapters/16-interaction.qmd b/chapters/14-interaction.qmd similarity index 100% rename from chapters/16-interaction.qmd rename to chapters/14-interaction.qmd diff --git a/chapters/17-missingness-and-measurement.qmd b/chapters/15-missingness-and-measurement.qmd similarity index 100% rename from chapters/17-missingness-and-measurement.qmd rename to chapters/15-missingness-and-measurement.qmd diff --git a/chapters/18-mediation.qmd b/chapters/16-mediation.qmd similarity index 100% rename from chapters/18-mediation.qmd rename to chapters/16-mediation.qmd diff --git a/chapters/19-longitudinal.qmd b/chapters/17-longitudinal.qmd similarity index 100% rename from chapters/19-longitudinal.qmd rename to chapters/17-longitudinal.qmd diff --git a/chapters/20-survival.qmd b/chapters/18-time-to-event.qmd similarity index 54% rename from chapters/20-survival.qmd rename to chapters/18-time-to-event.qmd index fe75956..5b0f607 100644 --- a/chapters/20-survival.qmd +++ b/chapters/18-time-to-event.qmd @@ -1,4 +1,4 @@ -# Causal survival analysis {#sec-survival} +# Causal time-to-event models {#sec-survival} {{< include 00-setup.qmd >}} @@ -8,7 +8,7 @@ status("unstarted") ``` -## Preparing data for survival analysis +## Preparing data for time-to-event analysis ```{r} rnorm(5) @@ -16,4 +16,4 @@ rnorm(5) ## Pooled logistic regression -## Confidence intervals for causal survival models +## Confidence intervals for causal time-to-event models diff --git a/chapters/21-sensitivity.qmd b/chapters/19-sensitivity.qmd similarity index 100% rename from chapters/21-sensitivity.qmd rename to chapters/19-sensitivity.qmd diff --git a/chapters/22-doubly-robust.qmd b/chapters/20-doubly-robust.qmd similarity index 100% rename from chapters/22-doubly-robust.qmd rename to chapters/20-doubly-robust.qmd diff --git a/chapters/22-machine-learning.qmd b/chapters/21-machine-learning.qmd similarity index 100% rename from chapters/22-machine-learning.qmd rename to chapters/21-machine-learning.qmd diff --git a/chapters/23-iv-and-friends.qmd b/chapters/22-iv-and-friends.qmd similarity index 100% rename from chapters/23-iv-and-friends.qmd rename to chapters/22-iv-and-friends.qmd