Problem set solutions should be written in Quarto Markdown (.qmd) source files, interspersing explanatory text with Python (and in some cases bash) code chunks. Please do not use Jupyter notebook (.ipynb) files as your underlying source file for your solutions. In some cases we will ask that you put function definitions for more complicated functions into one or more Python code (.py) file(s) and show us the code in the appendix of your main solution file by using inspect.getsource().
We’ll use data from several real world situations in class.
-By now you should already have access to the following 5 basic tools:
+Today we will use all these tools together to submit a solution for Problem Set 0 (not a real problem set) to make sure you know how to submit solutions to upcoming (real) problem sets.
+Here is a selection of some basic reference tutorials and documentation for unix, bash and unix commands, git & GitHub, quarto, python and VS Code
+ +Discussion: - Why is git so damn complicated? - What do you need to remember when working with collaborators on the same repository?
+Refer to this guide and please ask questions if something is not clear.
+If we finish early, We will also take today’s lab as an opportunity to get familiar with the basic use of all the 5 basic tools listed above.
+For git and quarto, very basic knowledge should be sufficient for now, but for unix commands and python, the more you learn the more effective you will be at solving the problem sets (and at any computational task you take on after that). You will need to learn more advanced use of git and github towards the end of the semester when you start working with other team members on the same project.
+Like RMarkdown, quarto allows for several execution options to be set per document and per chunk. Spend some time getting familiar with the various options, and keep this link handy when you are working on the first few problem sets.
+Depending on what’s required in the problem sets, you may need to set eval to false (just print out code) or error to true (print errors and don’t halt rendering of the document). Some of the other options may be useful for controlling how the code gets printed.
Install tinytex via quarto install tinytex
If you are using the knitr engine, you should be able to tag your code chunks in quarto with {bash} and use verbatim bash commands. If you are using the Jupyter engine, the {python} tag should be used instead and every line containing a bash command should be prefixed with an exclamation mark (!).
You can go to the settings in your IDE and point it to the specific python installation that you find when you execute which python in the terminal.
+you can fix the quarto configuration by setting the environment variable QUARTO_PYTHON to the correct python path or by running quarto check. Restarting the IDE may also help if you had just installed something in the other environment.
Generate a numpy array (named x) of random numbers from a standard normal distribution with 20 columns and as many rows as needed so that the data take up about 16 MB (megabytes) in size. As part of your answer, show the arithmetic (formatted using LaTeX math syntax) you did to determine the number of rows.
Explain the sizes of the two files created below. In discussing the CSV text file, how many characters do you expect to be in the file (i.e., you should be able to estimate this reasonably accurately from first principles without using wc or any explicit program that counts characters). Hint: what do we know about numbers drawn from a standard normal distribution?
import os
import pandas as pd
x = x.round(decimals = 12)
@@ -358,7 +358,7 @@ Problems
pd.DataFrame(x).to_pickle('x.pkl', compression = None)
print(f"{str(os.path.getsize('x.pkl')/1e6)} MB")30.777135 MB
+30.778352 MB
16.000572 MB