Merge branch 'main' of https://github.com/berkeley-stat243/stat243-fa…

…ll-2023 into main

ps/ps6.qmd

@@ -0,0 +1,93 @@
+---
+title: "Problem Set 6"
+subtitle: "Due Friday Nov. 3, 10 am"
+format:
+  pdf:
+    documentclass: article
+    margin-left: 30mm
+    margin-right: 30mm
+    toc: false
+  html:
+    theme: cosmo
+    css: ../styles.css
+    toc: false
+    code-copy: true
+    code-block-background: true
+execute:
+  freeze: auto
+---
+
+## Comments
+
+- This covers material in Units 8 and 9.
+- It's due at 10 am (Pacific) on November 3, both submitted as a PDF to Gradescope as well as committed to your GitHub repository.
+- Please see PS1 for formatting and attribution requirements.
+- Note that is is fine to hand-write solutions to the the non-coding questions, but make sure your writing is neat and insert any hand-written parts in order into your final submission. 
+
+
+## Problems
+
+1. In class and in the Unit 8 notes, I mentioned that integers as large as $2^{53}$ can be stored exactly in the double precision floating point representation. (Note that for this problem, you don't need to write out `e` in base 2; you can use base 10).
+   a. Demonstrate how the integers 1, 2, 3, ...,$2^{53}-2$, $2^{53}-1$ can be stored exactly in the $(-1)^{S}\times1.d\times2^{e-1023}$ format where d is represented as 52 bits. I'm not expecting anything particularly formal - just write out for a few numbers and show the pattern.
+   b. Then show that $2^{53}$ and $2^{53}+2$ can be represented exactly but $2^{53}+1$ cannot, so the spacing of numbers of this magnitude is 2. Finally show that for numbers starting with $2^{54}$ that the spacing between integers that can be represented exactly is 4.  Then confirm that what you've shown is consistent with the result of executing $2.0^{53}-1$, $2.0^{53}$, and $2.0^{53}+1$ in Python (you can use base Python floats or numpy).
+   c. Finally, calculate the relative error in representing numbers of magnitude $2^{53}$ in base 10. (This should, of course, look very familiar, and should be the same as the relative error for numbers of magnitude $2^{54}$ or any other magnitude...)
+
+2. If we want to estimate a derivative of a function on a computer
+    (often because it is hard to calculate the derivative analytically),
+    a standard way to approximate this is to compute:
+    $$f'(x)\approx\frac{f(x+\epsilon)-f(x)}{\epsilon}$$ for some small
+    $\epsilon$. Since the limit of the right-hand side of the expression
+    as $\epsilon\to0$ is exactly $f'(x)$ by the definition of the
+    derivative, we presumably want to use $\epsilon$ very small from the
+    perspective of using a difference to approximate the derivative.
+
+    a. Considering the numerator, if we try to do this on a computer, in what ways (there are more than
+    one) do the limitations of arithmetic on a computer affect our choice of $\epsilon$? (Note that I am ignoring the denominator because that just scales the magnitude of the result and itself has 16 digits of accuracy.)
+    b. Write a Python function that calculates the approximation and explore how the error in the estimated derivative for some $x$ varies as a function of $\epsilon$ for a (non-linear) function that you choose such that you can calculate the derivative analytically (so that you know the truth).
+
+
+3. Consider multiclass logistic regression, where you have
+    quantities like this:
+    $$p_{j}=\text{Prob}(y=j)=\frac{\exp(x\beta_{j})}{\sum_{k=1}^{K}\exp(x\beta_{k})}=\frac{\exp(z_{j})}{\sum_{k=1}^{K}\exp(z_{k})}$$
+    for $z_{k}=x\beta_{k}$. Here $p_j$ is the probability that the observation, $y$, is in class $j$.
+
+   a. What will happen if the $z$ values are very large in magnitude (either positive or negative)?
+   b. How can we reexpress the equation so as to be able to do the calculation even when either of those situations occurs?
+
+4. Let's consider importance sampling and explore the need to have the
+sampling density have heavier tails than the density of interest.
+Assume that we want to estimate $\phi=EX$ and $\phi=E(X^{2})$ with
+respect to a density, $f$. We'll make use of the Pareto distribution,
+which has the pdf $p(x)=\frac{\beta\alpha^{\beta}}{x^{\beta+1}}$
+for $\alpha<x<\infty$, $\alpha>0$, $\beta>0$. The mean is $\frac{\beta\alpha}{\beta-1}$
+for $\beta>1$ and non-existent for $\beta\leq1$ and the variance
+is $\frac{\beta\alpha^{2}}{(\beta-1)^{2}(\beta-2)}$ for $\beta>2$
+and non-existent otherwise.
+
+   a. Does the tail of the Pareto decay more quickly or more slowly than
+that of an exponential distribution?
+   b. Suppose $f$ is an exponential density with parameter value equal
+to 1, shifted by two to the right so that $f(x)=0$ for $x<2$. Pretend
+that you can't sample from $f$ and use importance sampling where
+our sampling density, $g$, is a Pareto distribution with $\alpha=2$
+and $\beta=3$. Use $m=10000$ to estimate $EX$ and $E(X^{2})$ and
+compare to the known expectations for the shifted exponential. Recall
+that $\mbox{Var}(\hat{\phi})\propto\mbox{Var}(h(X)f(X)/g(X))$. Create
+histograms of $h(x)f(x)/g(x)$ and of the weights $f(x)/g(x)$ to
+get an idea for whether $\mbox{Var}(\hat{\phi})$ is large. Note if
+there are any extreme weights that would have a very strong influence
+on $\hat{\phi}$.
+   c. Now suppose $f$ is the Pareto distribution described above and pretend
+you can't sample from $f$ and use importance sampling where our sampling
+density, $g$, is the exponential described above. Respond to the
+same questions as for part (b), comparing to the known values for
+the Pareto.
+
+
+5. Extra credit: This problem explores the smallest positive number that base Python or numpy can represent and how numbers just larger than the smallest positive number that can be represented. 
+
+   a. By trial and error, find the base 10 representation of the smallest positive number that can be represented in Python. Hint: it's rather smaller than $1\times10^{-308}$. 
+
+   b. Explain how it can be that we can store a number smaller than $1\times2^{-1022}$, which is the value of the smallest positive number that we discussed in class. Start by looking at the bit-wise representation of $1\times2^{-1022}$. What happens if you then figure out the natural representation of $1\times2^{-1023}$? You should see that what you get is actually a very "well-known" number that is not equal to $1\times2^{-1023}$. Given the actual bit-wise representation of $1\times2^{-1023}$, show the progression of numbers smaller than that that can be represented exactly and show the smallest number that can be represented in Python written in both base 2 and base 10.
+
+      Hint: you'll be working with numbers that are not normalized (i.e., denormalized); numbers that do not have 1 as the fixed number before the radix point in the floating point representation we discussed in Unit 8.

publish.md

@@ -14,6 +14,8 @@ To set up the website for a new course/year, one needs to run `quarto publish` f
 
 Notes:
 
+2023-10-25: Today (and on some past occasions), figures don't show up in the html because the relevant png files are not committed/being deleted from `units/unitX_files`. Manually adding to `gh-pages` works for a given commit but then later render/publish causes a `git rm` of the files. I think what may be the solution is to make sure not to do `git commit -am` if a `git rm` of the png files is staged. Instead just use `git add` on the various files and `git commit -m`. May also need to `git restore` the files to be deleted so that later `git commit -am'` doesn't cause deletion.
+
 2023-09-07: GHA can fail with messages about `nbformat`. Can often fix by re-rendering the problematic qmd. I think this is happening when commits are made to a qmd without rendering that updates the freeze files. 
 
 2023-08-29: when using `knitr` engine with `unit3-bash.qmd` (so that one can work with bash chunks), some GHA runs are complaining about missing `rmarkdown`. But then it sometimes works. Trying to install `rmarkdown` leads to a permission issue in the system directory that the R package is being installed into. If try to use `jupyter` engine with bash chunks, you probably need a Jupyter bash kernel, but I am still investigating.

schedule.md

@@ -9,6 +9,7 @@ Submit your solutions on Gradescope and (for problem sets but not assignments) v
 | PS 3 [(HTML)](ps/ps3.html) [(PDF)](ps/ps3.pdf) [(code)](ps/ps3start.py) | Wednesday Sep. 27 | 10 am | 
 | PS 4 [(HTML)](ps/ps4.html) [(PDF)](ps/ps4.pdf) | Monday Oct. 9 | 10 am | 
 | PS 5 [(HTML)](ps/ps5.html) [(PDF)](ps/ps5.pdf) | Friday Oct. 27 | 10 am | 
+| PS 6 [(HTML)](ps/ps6.html) [(PDF)](ps/ps6.pdf) | Friday Nov. 3 | 10 am | 
 
 
 Problem set solutions need to follow the rules discussed in Lab 1 (Sep. 1) and documented [here](howtos/ps-submission).