-
Notifications
You must be signed in to change notification settings - Fork 1
/
index.html
1889 lines (1818 loc) · 85.8 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>A/B Testing</title>
<meta name="author" content="James Ha">
<meta name="apple-mobile-web-app-capable" content="yes">
<meta name="apple-mobile-web-status-bar-style" content="black-translucent">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
<link rel="stylesheet" href="css/reveal.min.css">
<link rel="stylesheet" href="custom/css/foo.min.css">
<link rel="stylesheet" href="css/theme/custom.css" id="theme">
<!-- Theme used for syntax highlighting of code -->
<link rel="stylesheet" href="lib/css/zenburn.css">
<!-- Printing and PDF exports -->
<script>
var link = document.createElement('link');
link.rel = 'stylesheet';
link.type = 'text/css';
link.href = window.location.search.match(/print-pdf/gi) ? 'css/print/pdf.css' : 'css/print/paper.css';
document.getElementsByTagName('head')[0].appendChild(link);
</script>
</head>
<body>
<div class="reveal">
<div class="slides">
<section
data-transition="fade-in">
<h1>A/B Testing</h1>
<ul>
<li class="fragment">Controlled experiments to determine a causal relationship between changes to an
application and their influence on observable behavior.
</li>
</ul>
<aside class="notes">
<p>These are my notes</p>
</aside>
</section>
<section
data-transition="fade-in">
<h2>Kinds of A/B Tests</h2>
<ul>
<li class="fragment">Split Tests, a.k.a. <strong>Hpothesis Testing</strong></li>
<li class="fragment"><strong>Multivariate Tests</strong>
<ul>
<li><em>Which really aren't A/B tests, but get grouped in with them anyway</em></li>
</ul>
</li>
<li class="fragment"><strong>A/A Tests</strong> - to make sure the service isn’t fake news</li>
<li class="fragment">Bayesian craziness, a.k.a. <strong>Simulations</strong></li>
</ul>
<div class="slide-footer-left">
<p class="footer-text">Introduction</p>
</div>
</section>
<section
data-transition="fade-in">
<h2>Yeah, but why?</h2>
<ul>
<li>Improves User Experience</li>
<li>Accelerates Innovation</li>
<li><strong style="font-family: didot;">CASH MONEY $$$ SYNERGY ROI</strong></li>
<li><strong style="font-family: didot;">THE BLOCKCHAIN</strong></li>
</ul>
<div class="slide-footer-left">
<p class="footer-text">Introduction</p>
</div>
<aside class="notes" data-markdown>
* **Improves User Experience** - we pick the variant that users prefer
* **Accelerates Innovation** - instead of wondering for ourselves which variant is better, we just test, and then have data to back up our choices
* Do you know that part on Amazon where it says, "people who bought this item also bought...."
* That part is the result of an A/B Test, implemented by a guy named Greg Linden.
* Actually, marketing senior vice president was very against it because he thought adding more additional information would overload the user when the use should be focused on checking out.
* So Greg Linden stuck the code in there in an A/B Test
* It turns out that implementing that feature made them a ton of money
</aside>
</section>
<section
data-transition="fade-in">
<h2>Concerns in A/B Tests</h2>
<ul>
<li class="fragment">How can I measure and analyze my test?</li>
<li class="fragment">How big does my test need to be?</li>
<li class="fragment">What can I test?</li>
<li class="fragment">What counts as a test?</li>
<li class="fragment">What are "statistics?"</li>
<li class="fragment">What is "probability?"</li>
</ul>
<div class="slide-footer-left">
<p class="footer-text">Introduction</p>
</div>
<aside class="notes" data-markdown>
* If the test is too small, the data might just be noise.
* If the test is too big, we will move too slowly
</aside>
</section>
<section
data-transition="fade-in">
<h2>Presentation Overview</h2>
<ol>
<li>Define Probability and Statistics</li>
<li>Frequentist Statistics</li>
<li>Bayesian Statistics</li>
<li><span style="text-decoration: line-through;">Multivariate Testing</span></li>
<li>Implementation</li>
<li>Faults in A/B Tests</li>
<li>A picture of a red panda</li>
</ol>
<div class="slide-footer-left">
<p class="footer-text">Introduction</p>
</div>
<aside class="notes" data-markdown>
* Sorry, I'm gonna skip multivariate testing (or at least the math for it) because it is nuts
complicated
* Everything that makes A/B testing complicated is orders of magnitude crazier when you introduce
multi-dimensional parameters
* Towards the end of part of Bayesian statics I'll mention briefly a family of tools called Markov Chain
Monte Carlo simulations, which is one of the major elements of multivariate testing.
</aside>
</section>
<section
data-transition="fade-in">
<h1>1. Define Probability & Statistics</h1>
<video data-autoplay loop>
<source data-src="custom/media/adventure-time.mp4" type="video/mp4"/>
</video>
</section>
<section
data-background-video="custom/media/adventure-time.mp4"
data-background-video-loop="true"
data-transition="fade-in">
<aside class="notes">
<ul>
<li>So I'm gonna try to cram a bunch of statistics in the next hour</li>
<li>I really do think it's important to understand why or how decisions in A/B testing are made.
</li>
<li>We could skip all this and just say, okay show me the formulas and I'll plug the A/B test data
in
</li>
<li>I hope someday we can implement that for our applications, but it's worth noting that very
critical business and technical decisions can be made over A/B testing analysis, we have to be
sure of the reasoning
</li>
<li>I'm gonna throw a lot of math at you guys and it's not too important to memorize the formulas or
proofs, but getting the general concepts and approaches to thinking about statistics is key.
</li>
<li>I know there might be a lot of questions, so I guess stop me if I totally lose you.</li>
</ul>
</aside>
</section>
<section
data-transition="fade-in">
<h2>What’s the point?</h2>
<ol>
<li>If I flip a coin 10 times and it lands heads 7 times, is the coin rigged? \(P(X = 7)\)</li>
<li class="fragment">10% of contributors who debrief a session will create an annotation.
<ul>
<li>A change is made to the application, and 8 of the next 50 debriefing contributors create
annotations.
</li>
<li>Did the change have an effect?</li>
<li>Is a sample size \(n=50\) good enough?</li>
</ul>
</ol>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes" data-markdown>
* If you can solve for the first problem, you are well on your way to solve the second problem
* This will be a pattern for most of the talk.
* I'll introduce an example using coin flips or dice throws
* And we'll use the solution to those examples to solve real A/B testing problems
</aside>
</section>
<section
data-transition="fade-in">
<h2>Probability 101</h2>
<ul>
<li>What is Probability \((P)\)? A measure of the likelihood an event will occur.</em></li>
<li>\(P(A)\) - The probability of event A</li>
<li>\(p=0\) - the event will not occur</li>
<li>\(p=1\) - the event will occur</li>
<li>\(P(X = x) = p(x)\)</li>
</ul>
<aside class="notes">
<ul>
<li>Depending on which textbook or paper you read, different authors use different notations.</li>
<li>The notation I'll be using will the ones I see most commonly, and I'll try to consistent</li>
<li>Some people write the variance symbole as sigma squared, and some use capital V-A-R</li>
<li>Probability of an event is capital P</li>
<li>Events are always uppercase letters</li>
<li>Lower case p is the probability function, or if you can express the probability of something as
a formula
</li>
</ul>
</aside>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<h2>Conditional Probability</h2>
<ul>
<li>
<strong>Conditional Probability</strong> \(P(A|B)\) - the probability of event A occurring, given B
has occurred
<span class="math-formula">\(P(A|B) = \frac{P(A \cap B)}{P(B)}\)</span>
</li>
<li>
<strong>Multiplication Rule</strong>
<span class="math-formula">\(P(A \cap B) = P(A|B) \cdot P(B)\)</span>
</li>
<li>
<p><strong>Mutually Exclusive</strong></p>
<span class="math-formula">\(P(A \cap B) = P(A) \times P(B)\)</span>
</li>
</ul>
<aside class="notes">
<ul>
<li>upside-down U is intersection</li>
</ul>
</aside>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<h2>Conditional Probability</h2>
<p>What is the probability that you draw from a pack of cards 2 diamonds in a row?</p>
<span class="math-formula fragment">\(P(D_2 \cap D_1)\)</span>
<span class="math-formula fragment">= \(P(D_2|D_1) \times P(D_1)\)</span>
<div class="fragment">
<p class="math-formula">\(P(D_1) = \frac{13}{52} = \frac{1}{4}\)</p>
<p class="math-formula">\(P(D_2|D_1) = \frac{12}{51}\)</p>
</div>
<p class="math-formula fragment">\(P(D_2 \cap D_1) = \frac{1}{4} * \frac{12}{51} = \frac{3}{51}\)</p>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes">
<ul>
<li>Ahem, What is the probability that the second card drawn from a pack of cards is a diamond
<span>\(P(D_2 \cap D_1)\)</span>, given that the first card drawn was a diamond P(D1)?
</li>
<li>Probability of drawing the first diamond is 13/52</li>
<li>Since there is 1 less card, the probability of drawing the second diamond given that a diamond
has been drawn is 12/51
</li>
</ul>
</aside>
</section>
<section
data-transition="fade-in">
<h2>Sally Clark</h2>
<p></p>
<ul>
<li><em>What is the probability that 2 babies from one mother die of SIDS?</em></li>
<li><em>Are these events dependent or independent?</em></li>
<li class="fragment">Given that a mother's first baby died of SIDS, what is the probability that her
second baby will also die of SIDS? \(P(B_2|B_1)\)
</li>
</ul>
<ul class="fragment">
<li>Expert testimony from Dr. Roy Meadow: \(P(B)^2 = (1/8543)^2 \approx 1/73000000\)</li>
<li>Convicted in Nov. 1999 on this evidence</li>
</ul>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside data-markdown class="notes">
* What is the probability that a mother's first 2 babies dies of SIDS? Are these events dependent or
independent?
* Given that a mother's first baby died of SIDS, what is the probability that her second baby will also
die of SIDS?
* Professor Sir Roy Meadow incorrectly concluded that these were 2 independent events, and famously said
it was a "1 in 73 million" chance.
* This false statistic lead to her false conviction in November 1999.
* Probability and statistics got distorted - people understood "1 in 73 million" to be a measure of her
innocence.
* Also the pathologist hid knowledge that one of the babies had a staph infection
* She was release on January 2003, but died in March 2007 from alcohol poisoning.
</aside>
</section>
<section
data-transition="fade-in">
<div>
<h2>Discrete Random Variables</h2>
<ul>
<li><strong>Random Variables</strong> - variable with a value determined by a chance event.</li>
<li>A <strong>discrete sample space</strong> \(\Omega\) is a finite set of outcomes \(\{\omega_1,
\omega_2...*\}\). The probability of outcome \(\omega\) is \(P(\omega)\).
</li>
</ul>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<div data-markdown>
## Discrete Probability Distribution
*a distribution that can be represented by a table.*
</div>
<div class="fragment">
<div data-markdown>
* Example: there are only 4 outcomes from flipping a coin 2 times.
</div>
<table>
<tr>
<th>Number of Heads, \(x\)</th>
<th>Probability \(P(x)\)</th>
</tr>
<tr>
<td>0</td>
<td>0.25</td>
</tr>
<tr>
<td>1</td>
<td>0.5</td>
</tr>
<tr>
<td>2</td>
<td>0.25</td>
</tr>
</table>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes" data-markdown>
* Note that the sum of all these probabilities equals 0.
</aside>
</section>
<section
data-transition="fade-in">
<div>
<h2>Discrete Functions</h2>
<ul>
<li><strong>Probability Mass Function pmf</strong> - the function for a discrete random
variable<br/>
<span class="math-formula">$$p(a) = P(X = a)$$ $$0 \le p(a) \le 1$$</span>
</li>
<li><strong>Cumulative Distribution Function cdf</strong> - the function that gives the total
probabilities from \(\infty\) to \(a\)<br/>
<span class="math-formula">$$F(a) = P(X \le a)$$</span>
</li>
</ul>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes" data-markdown>
* You can often determine which one you need by listening to the question.
* If someone asks what is the probability of getting exactly blah heads in so many coin flips, it's a
pmf
* If someone asks what is the probability of getting at least blah head in so many coin flips, it's a
cdf
</aside>
</section>
<section
data-transition="fade-in">
<div>
<h2>Example: Discrete Functions</h2>
<ul>
<li>Let the sample space \(\Omega\) be 2 dice rolls</li>
<li>Random variable \(M\) is the maximum of 2 dice rolls.
<span class="math-formula">$$M(1,4) = 4$$</span>
</li>
</ul>
<div class="fragment">
<table>
<tr>
<th>value</th>
<th>\(a\)</th>
<th>1</th>
<th>2</th>
<th>3</th>
<th>4</th>
<th>5</th>
<th>6</th>
</tr>
<tr>
<td>pmf</td>
<td>\(p(a)\)</em></td>
<td>1/36</td>
<td>3/36</td>
<td>5/36</td>
<td>7/36</td>
<td>9/36</td>
<td>11/36</td>
</tr>
<tr>
<td>cdf</td>
<td>\(F(a)\)</td>
<td>1/36</td>
<td>4/36</td>
<td>9/36</td>
<td>16/36</td>
<td>25/36</td>
<td>36/36</td>
</tr>
</table>
</div>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<div>
<h2>Discrete Mean & Variance</h2>
<div>
<ul>
<li><strong>Discrete Mean</strong> - the average of a discrete random variable \(X\) a.k.a.
<strong>expected value</strong> of \(X\)
<span class="math-formula">$$E(X) = \mu_{x} = \sum_{i=1}^n p(x_{i}){x_{i} }$$</span>
</li>
</ul>
</div>
<div class="fragment">
<ul>
<li><strong>Discrete Variance</strong> - a measure of how much the probability mass is spread
out around \(\mu\).
<span class="math-formula">$$Var(X) = E((X- \mu )^{2}) = \sum_{i=1}^n p({x_{i})(x_{i}- \mu )^{2} }$$</span>
</li>
</ul>
</div>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<div>
<h2>Discrete Standard Deviation</h2>
<div>
<ul>
<li><strong>Discrete Standard Deviation \(\sigma\)</strong> - a measure of the spread, expressed
in the same units as the expected Value
<span class="math-formula">$$\sigma = \sqrt{Var(X)}$$</span>
<span class="math-formula fragment">$$Var(X) = \sigma^2$$</span></li>
</ul>
</div>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes">
<ul>
<li>Expected Value is written as capital E of x</li>
<li>Mean is usually written with lowercase Greek μ</li>
</ul>
</aside>
</section>
<section
data-transition="fade-in">
<div data-markdown>
## Bernoulli Trial
* An experiment that only has two possible results - *success* and *failure* - is a **Bernoulli Trial**
if:
* The results are mutually exclusive,
* The probability of these two results do not change each time the experiment is done
</div>
<span class="math-formula">$$X =\begin{cases}1 & success\\0 & failure\end{cases}$$</span>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<div>
<h2>Bernoulli Distribution</h2>
<ul>
<li>Success:<br/>
<span class="math-formula">\(P(X = 1) = p\)</span>
</li>
<li>Failure:<br/>
<span class="math-formula">\(P(X = 0) = 1-p\)</span>
</li>
<li>Expected Value:<br/>
<span class="math-formula">\(E(X) = \mu = 0(1-p) + 1(p) = p\)</span>
</li>
<li>Variance:<br/>
<span class="math-formula">\(\sigma_x^2 = p(1-p)\)</span>
</li>
</ul>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes">
<ul>
<li>If we say a success is rolling a die for 3 or higher, then the probability of success is 0.67,
and the probability of failure is 0.33.
</li>
</ul>
</aside>
</section>
<section
data-transition="fade-in">
<div>
<h2>Binomial Distribution</h2>
<p><em>Lots and lots of Bernoulli Trials</em></p>
<ul class="fragment">
<li><strong>Binomial Coefficient</strong> - the number of ways to “choose” \(k\)
unordered outcomes from \(n\) possibilities
<span class="math-formula">$$_nC_k = \begin{pmatrix}n \\k \end{pmatrix} = \frac{n!}{k!(n - k)!}$$</span>
</li>
<li class="fragment">Binomial probability mass function:
<span class="math-formula">$$P(X = k) = \begin{pmatrix}n \\k \end{pmatrix}p^{k}(1-p)^{n-k}$$</span>
</li>
</ul>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes" data-markdown>
* A bernoulli trial is one user coming to your website, he either buys or he doesn't buy
* A binomial distribution is 1000 users coming to your site. Some buy and some don't buy
* For example, how many ways can you choose 2 heads from flipping 4 coins? it would be 4! divided by 2!
times 2! which equals 6
</aside>
</section>
<section
data-transition="fade-in">
<div>
<h2>Binomial Math</h2>
<ul>
<li>Mean:<br/>
<span class="math-formula">\(\mu_x = np\)</span>
</li>
<li>Variance:<br/>
<span class="math-formula">\(\sigma_x^2 = np(1-p)\)</span>
</li>
<li>Standard deviation:<br/>
<span class="math-formula">\(\sigma_x = \sqrt{np(1-p)}\)</span>
</li>
</ul>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes" data-markdown>
* So if I have 100 coins and they are all fair, the expected value of heads is 100 * 0.5 = 50
</aside>
</section>
<section
data-transition="fade-in">
<div>
<h2>Binomial Coins</h2>
<p><em>What is the probability of getting 3 heads in 5 coin flips?</em></p>
<span class="math-formula fragment">$${P(X = 3) = \begin{pmatrix}5 \\3 \end{pmatrix}.5^{3}(1-0.5)^2 = 31.25\%}$$</span>
<p class="fragment"><em>What is the cdf of 3 out of 5?</em></p>
<table class="fragment" style="font-size: 75%;">
<tr>
<th>Value</th>
<th>\(a\)</th>
<th>0</th>
<th>1</th>
<th>2</th>
<th>3</th>
<th>4</th>
<th>5</th>
</tr>
<tr>
<td>pmf</td>
<td>\(p(a)\)</td>
<td>\(p^5\)</td>
<td>\(5p^5\)</td>
<td>\(10p^5\)</td>
<td>\(10p^5\)</td>
<td>\(5p^5\)</td>
<td>\(p^5\)</td>
</tr>
<tr>
<td>pmf</td>
<td>\(p(a)\)</td>
<td>\(0.03125\)</td>
<td>\(0.15625\)</td>
<td>\(0.3125\)</td>
<td>\(0.3125\)</td>
<td>\(0.15625\)</td>
<td>\(0.03125\)</td>
</tr>
<tr>
<td>cdf</td>
<td>\(F(a)\)</td>
<td>\(0.03125\)</td>
<td>\(0.1875\)</td>
<td>\(0.5\)</td>
<td>\(0.8125\)</td>
<td>\(0.99875\)</td>
<td>\(1\)</td>
</tr>
</table>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<div>
<h2>Binomial Views</h2>
<p><em>90% of students will view their scores if released to them.</em></p>
<p><em>What's the probability all of a class of 20 will view?</em></p>
<span class="math-formula fragment">$${P(X = 20) = \begin{pmatrix}20 \\20 \end{pmatrix}.9^{20}(1-.9)^0 = 12.16\%}$$</span>
<p class="fragment"><em>What's the probability at least 90% will view?</em></p>
<span class="fragment">$${P(X \ge 18) = 1 - P(X=20 )- P(X=19 ) = 60.83\%}$$</span>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes">
<ul>
<li>Like the coin flipping table, we can add up all the probabilites of 0 students, 1 student, 2
students but that takes forever
</li>
<li>Instead take advantage that the sum of all probabilities always equal 1</li>
<li>It's faster do 1 minus the probabilities of 20 students and 19 students</li>
</ul>
</aside>
</section>
<section
data-transition="fade-in">
<div>
<h2>Binomial Views With Code</h2>
<p><em>Ain’t nobody got time to math.</em></p>
<pre><code data-trim data-noescape>
from scipy.stats import binom
sample_size = 20
p = 0.9
# all 20 will view score
all_probability = binom.pmf(20, sample_size, p)
print(all_probability)
# 0.121576654591
# at least 18 students will view score
cumulative_probability = binom.cdf(20, sample_size, p)
print(cumulative_probability)
# 0.608253001875
</code></pre>
</div>
</section>
<section
data-transition="fade-in">
<div>
<h2>Continuous Random Variables</h2>
<ul>
<li>A random variable \(X\) is <strong>continuous</strong> if there is a function \(f(x)\) such that
for any \(c \le d\), the probability density function (pdf) is
<span class="math-formula">$$P(c \leq d) = \int_c^d f(x)dx$$</span>
<span class="math-formula">$${P(-\infty \leq X \leq \infty) = \int_{-\infty}^\infty f(x) dx = 1}$$</span>
</li>
</ul>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes">
<ul>
<li>So note that when dealing with discrete distributions we are looking for a probability mass
function.
</li>
<li>For continuous distribution, we are looking for a probability density function.</li>
<li>Mass is the integral of density</li>
<li>If you take the integral of the pdf at all vales, then the area of all probability is 1</li>
</ul>
</aside>
</section>
<section
data-transition="fade-in">
<div>
<h2>Continuous Distributions</h2>
<table>
<tr>
<td>Uniform</td>
<td>angle of a dart throw</td>
<td>\(U(a,b)\)</td>
</tr>
<tr>
<td>Beta</td>
<td>batting average</td>
<td>\(beta(a,b)\)</td>
</tr>
<tr>
<td>Exponential</td>
<td>Finding an Uber</td>
<td>\(exp(\lambda)\)</td>
</tr>
<tr>
<td>Normal</td>
<td>IQ</td>
<td>\(N(\mu,\sigma^2)\)</td>
</tr>
<tr>
<td>Student \(t\)</td>
<td>Guinness Beer</td>
<td>\(t(df)\)</td>
</tr>
</table>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<h2>The Boring Distribution</h2>
<p><em>a.k.a “Bell Curve”, Gaussian Distribution</em></p>
<span class="math-formula">$$f(x) = \phi(z) = \frac{1}{\sigma \sqrt{2 \pi } }e^{\frac{-(x - \mu)^2}{2\sigma^2}}$$</span>
<ul class="fragment">
<li>Defined by mean and standard deviation \(N(\mu,\sigma^2)\)</li>
<li>Curve is symmetrical, more data at center vs. edges</li>
<li>Mean = Median = Mode</li>
<li>Don't assume everything is normally distributed</li>
</ul>
</section>
<section
data-transition="fade-in">
<h2>Standard Z</h2>
<p><em>A normal distribution where \(\mu\ = 0\), \(\sigma = 1\), a.k.a. \(N(0,1)\)</em></p>
<span class="math-formula">$$Z = \frac{X - \mu }{ \sigma }$$</span>
<ul class="fragment">
<li>If \(X=1\), then it is 1 standard deviation from the mean</li>
<li>pdf: \(f(z) = \phi(z) = 1/2\)</li>
<li>\(P(-1 \le Z \le 1) \approx 0.6827\)</li>
<li>\(P(-2 \le Z \le 2) \approx 0.9545\)</li>
<li>\(P(-3 \le Z \le 3) \approx 0.9973\)</li>
</ul>
<aside class="notes">
<ul>
<li>Draw out the percentages!</li>
<li>The probability that Z is between -1 and 1 is 68%</li>
</ul>
</aside>
</section>
<section
data-transition="fade-in">
<h2>Expected Value of Continuous Distributions</h2>
<p><em>Expected Value measures central tendency</em></p>
<span class="math-formula">$$E(X) = \int_a^b xf(x)dx$$</span>
<div class="fragment">
<p>Expected Value of a standard z distribution</p>
<span class="math-formula">$$E(z) = \phi(z) = \frac{1}{ \sqrt{2 \pi } }e^{-z^2/2} \Big|_{-\infty}^{\infty} = 0$$</span>
</div>
</section>
<section
data-transition="fade-in">
<h2>Statistics vs. Probability</h2>
<ul>
<li><strong>Statistic</strong>
<ul>
<li><em>μ</em> of 100 dice rolls</li>
<li>Number times rolling a 5</li>
</ul>
</li>
<li><strong>Probability</strong>
<ul>
<li>Likelihood of rolling a 4</li>
<li>Likelihood of rolling a 6 three times</li>
</ul>
</li>
</ul>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<!--
<section
data-transition="fade-in">
<div data-markdown>
## Statistics
* What is **Statistic**? Information that can be computed from data
* **Pure Statistics** - a single value computed from data, such as a sample average
* **Interval Statistics** - an interval [*a, b*] computed from data.
* **Statistical Inference** - draw conclusions about a large data set by analyzing smaller sets of data
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
-->
<section
data-transition="fade-in">
<div data-markdown>
## Inferential Statistics
* **Parameter estimation** - some value that determines the properties of the distribution, such as
\\(\mu\\) or \\(\sigma\\)
* **Data Prediction** - use information about sample to predict a random selection
* **Model comparison** - selecting a model which best explains the observed data, something that
postulates the relationship between factors and the data
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<div>
<h2>Central Limit Theorem</h2>
<p><em>Without it, we can't make math happen!</em></p>
<span class="math-formula">$$\lim_{n \rightarrow \infty} P(| \overline{X} - \mu | < a) = 1$$</span>
<ul>
<li>The bigger the sample, the closer the mean and variance of the sample gets to the population
</li>
<li>The means of many samples is a normal distribution</li>
<li>The means of those means approximates the mean of the population</li>
</ul>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes">
<ul>
<li>The normal distribution just doesn't happen everywhere, contrary to what we think</li>
<li>Suppose you're researching penguins. Today you grab 10 penguins and get their average weight.
Tomorrow you grab another 10 penguins and get their average weight. You do this over an over
again and if you graph all those averages, you get a normal distribution.
</li>
<li>Experimentation is what makes the normal distribution</li>
</ul>
</aside>
</section>
<!--
<section
data-transition="fade-in">
<div>
<h2>Sample vs Population</h2>
<ul>
<li>Let \(S_n\) the the sum of \(x_1, x_2, ..., x_n\) random variables</li>
<li>Each has a mean \(\mu\) and a standard deviation \(\sigma\).</li>
<li>Then the weighted average of the random variables is:</li>
</ul>
<span class="math-formula">$$\overline{X}_{n} = \frac{S_{n}}{n} = \frac{X_{1} + ... + X_{n}}{n} = \big(\sum_{i=1}^nX_{i}\big)/n$$</span>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
-->
<section
data-transition="fade-in">
<div>
<h2>Sample Statistics</h2>
<p><em>Let \(S_n\) the the sum of \(x_1, x_2, ..., x_n\) random variables</em></p>
<table>
<tr>
<td>Mean</td>
<td>\(E( \overline{X}_{n}) =\mu\)</td>
<td>\(E(S_{n}) = n\mu\)</td>
</tr>
<tr>
<td>Variance</td>
<td>\(Var( \overline{X}_{n}) =\frac{\sigma^2}{n}\)</td>
<td>\(Var(S_{n}) = n\sigma^2\)</td>
</tr>
<tr>
<td>S.D.</td>
<td>\(\sigma_{\overline{X}_{n}} =\frac{\sigma}{\sqrt{n}}\)</td>
<td>\(\sigma_{S_{n}} =\sqrt{n}\sigma\)</td>
</tr>
</table>
<p class="fragment">As \(n\) gets larger, the standard deviation gets smaller</p>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<h2>Theory Time!<br/>Bayesian vs. Frequentist</h2>
<p><em>Statistics require making lots of inferences</em></p>
<ul>
<li class="fragment"><strong>Bayesian</strong>: probability is subjective and deduced from prior
knowledge
</li>
<li class="fragment"><strong>Frequentist</strong>: probability is objective and obtained through
experimentation
</li>
</ul>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
<aside class="notes">
<ul>
<li>So honestly, when you ask "what is Probability?" you can't even get a unified answer.</li>
<li>The reason is statistics require dealing a lot of unknowns. The process by which we we obtain
knowledge from a unknown information has two main approaches
</li>
<li>I'm going to give you a very simplified definition. If a real statistician were sitting here,
that person would cringe a little for trying compress an entire discipline into a few sentences.
</li>
<li>The first school of thought is Bayesian Statistics, which was popular until the 20th century
</li>
<li>Bayesian: probability is subjective and deduced from prior knowledge</li>
<li>The second school of thought is Frequentist Statistics, which is the predominant approach to
statistics since the 20th century.
</li>
<li>Frequentist: probability is objective obtained from experimentation</li>
<li>The funny thing is, in this 21st century, Bayesian inference is making a huge comeback with
regards to machine learning and data analysis
</li>
<li>The main point to remember is neither one is inherently wrong or superior, it's just that the
two have different goals when computing data.
</li>
</ul>
</aside>
</section>
<section
data-transition="fade-in">
<div>
<h2>Bayesian vs Frequentist</h2>
<p><em>A coin is flipped 100 times and it lands heads 60 times.</em></p>
<ul>
<li><strong>Frequentist</strong>: What's the probability of getting 60 or more heads \(P(X > 59)\)
if the coin is fair?
</li>
<li><strong>Bayesian</strong>: Is the coin fair?</li>
</ul>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<div>
<h2>What about A/B Testing?</h2>
<ul>
<li><strong>Frequentist</strong>: Do the results of the new variant support rejecting the current
variant?
</li>
<li><strong>Bayesian</strong>: What's the probability the new variant is better than the current
variant?
</li>
</ul>
</div>
<div class="slide-footer-left">
<p class="footer-text">1. Probability & Statistics</p>
</div>
</section>
<section
data-transition="fade-in">
<div class="full-page-background"
style="background-image: url('custom/media/xkcd-1132.png');height: 680px;"></div>
<aside class="notes" data-markdown>
* Of course there's a relevant XKCD. This one is number 1132
</aside>
</section>
<section
data-transition="fade-in">
<h1>Frequentist Statistics</h1>
<p><em>Probabilities present long-term frequencies of repeatable random experiments.</em></p>
<ul>
<li>Conclusions are objective</li>
<li>Approach problems like a flowchart</li>
<li>Assign probability to the data, not degrees of belief</li>
<li>Dominant approach for 20th century</li>
</ul>
<div class="slide-footer-left">
<p class="footer-text">2. Frequentist Statistics</p>