->data2
->1
->2
->… +> data1
+> data2
+> 1
+> 2
+> … You will then see a weird character or possibly the string `CR` or `\r`:
-thus causing terrible things to happen to your data. For example, `2\r` is not a valid integer, and thus will throw an error (if you’re lucky) when you attempt to operate on it in R or Python. Note that this happens on Excel for OSX as well as Windows, due to legacy Windows compatibility.
+thus causing terrible things to happen to your data. For example, `2\r` is not a valid integer, and thus will throw an error (if you're lucky) when you attempt to operate on it in R or Python. Note that this happens on Excel for OSX as well as Windows, due to legacy Windows compatibility.
There are a handful of solutions for enforcing uniform UNIX-style line endings on your exported CSV files:
-1. When exporting from Excel, save as a “Windows comma separated (.csv)” file
+1. When exporting from Excel, save as a "Windows comma separated (.csv)" file
+
2. If you store your data file under version control using Git, edit the `.git/config` file in your repository to automatically translate `\r\n` line endings into `\n`.
-Add the following to the file ([see the detailed tutorial](http://nicercode.github.io/blog/2013-04-30-excel-and-line-endings)):
-
- [filter "cr"]
- clean = LC_CTYPE=C awk '{printf(\"%s\\n\", $0)}' | LC_CTYPE=C tr '\\r' '\\n'
- smudge = tr '\\n' '\\r'`
-
- and then create a file `.gitattributes` that contains the line:
-
- *.csv filter=cr
-
-
-3. Use [dos2unix](http://dos2unix.sourceforge.net/) (available on OSX, *nix, and Cygwin) on local files to standardize line endings.
+ Add the following to the file ([see the detailed tutorial](https://nicercode.github.io/blog/2013-04-30-excel-and-line-endings)):
+
+ ```
+ [filter "cr"]
+ clean = LC_CTYPE=C awk '{printf(\"%s\\n\", $0)}' | LC_CTYPE=C tr '\\r' '\\n'
+ smudge = tr '\\n' '\\r'`
+ ```
+
+ and then create a file `.gitattributes` that contains the line:
+
+ ```
+ *.csv filter=cr
+ ```
+
+3. Use [dos2unix](https://dos2unix.sourceforge.net/) (available on OSX, \*nix, and Cygwin) on local files to standardize line endings.
#### A note on R and `.xlsx`
@@ -121,11 +130,20 @@ worksheets in the `.xlsx` documents.
#### Caveats on commas
-In some datasets, the data values themselves may include commas (,). This is particularly true in countries that
+In some datasets, the data values themselves may include commas (,). This is particularly true in countries that
use commas as decimal separators. In that case, the software which you use (including Excel)
will most likely incorrectly display the data in columns. This is because the commas which are a part of the data values will be
-interpreted as delimiters.
+interpreted as delimiters.
If you are working with data that contains commas, you likely will need to use another delimiter when working in a spreadsheet. In this
case, consider using tabs as your delimiter and working with TSV files. TSV files can be exported from spreadsheet
-programs in the same way as CSV files. For more of a discussion on data formats and potential issues with commas within datasets see [the discussion page](http://www.datacarpentry.org/spreadsheet-ecology-lesson/discuss/).
+programs in the same way as CSV files. For more of a discussion on data formats and potential issues with commas within datasets see [the discussion page](https://www.datacarpentry.org/spreadsheet-ecology-lesson/discuss/).
+
+:::::::::::::::::::::::::::::::::::::::: keypoints
+
+- Data stored in common spreadsheet formats will often not be read correctly into data analysis software, introducing errors into your data.
+- Exporting data from spreadsheets to formats like CSV or TSV puts it in a format that can be used consistently by most programs.
+
+::::::::::::::::::::::::::::::::::::::::::::::::::
+
+
diff --git a/data/survey_data_spreadsheet_messy.xls b/episodes/data/survey_data_spreadsheet_messy.xls
similarity index 100%
rename from data/survey_data_spreadsheet_messy.xls
rename to episodes/data/survey_data_spreadsheet_messy.xls
diff --git a/data/survey_sorting_exercise.xlsx b/episodes/data/survey_sorting_exercise.xlsx
similarity index 100%
rename from data/survey_sorting_exercise.xlsx
rename to episodes/data/survey_sorting_exercise.xlsx
diff --git a/fig/1_helpful_clippy.jpg b/episodes/fig/1_helpful_clippy.jpg
similarity index 100%
rename from fig/1_helpful_clippy.jpg
rename to episodes/fig/1_helpful_clippy.jpg
diff --git a/fig/2_datasheet_example.jpg b/episodes/fig/2_datasheet_example.jpg
similarity index 100%
rename from fig/2_datasheet_example.jpg
rename to episodes/fig/2_datasheet_example.jpg
diff --git a/fig/3_white_table_1.jpg b/episodes/fig/3_white_table_1.jpg
similarity index 100%
rename from fig/3_white_table_1.jpg
rename to episodes/fig/3_white_table_1.jpg
diff --git a/fig/4_merged_cells.jpg b/episodes/fig/4_merged_cells.jpg
similarity index 100%
rename from fig/4_merged_cells.jpg
rename to episodes/fig/4_merged_cells.jpg
diff --git a/fig/5_excel_dates_1.jpg b/episodes/fig/5_excel_dates_1.jpg
similarity index 100%
rename from fig/5_excel_dates_1.jpg
rename to episodes/fig/5_excel_dates_1.jpg
diff --git a/fig/6_excel_dates_2.jpg b/episodes/fig/6_excel_dates_2.jpg
similarity index 100%
rename from fig/6_excel_dates_2.jpg
rename to episodes/fig/6_excel_dates_2.jpg
diff --git a/fig/7_excel_dates_3.jpg b/episodes/fig/7_excel_dates_3.jpg
similarity index 100%
rename from fig/7_excel_dates_3.jpg
rename to episodes/fig/7_excel_dates_3.jpg
diff --git a/img/DC1_logo_small.png b/episodes/fig/DC1_logo_small.png
similarity index 100%
rename from img/DC1_logo_small.png
rename to episodes/fig/DC1_logo_small.png
diff --git a/img/DataONE_LOGO.jpg b/episodes/fig/DataONE_LOGO.jpg
similarity index 100%
rename from img/DataONE_LOGO.jpg
rename to episodes/fig/DataONE_LOGO.jpg
diff --git a/fig/conditional_formating.png b/episodes/fig/conditional_formating.png
similarity index 100%
rename from fig/conditional_formating.png
rename to episodes/fig/conditional_formating.png
diff --git a/img/creative-commons-attribution-license.png b/episodes/fig/creative-commons-attribution-license.png
similarity index 100%
rename from img/creative-commons-attribution-license.png
rename to episodes/fig/creative-commons-attribution-license.png
diff --git a/fig/csv-mistake.png b/episodes/fig/csv-mistake.png
similarity index 100%
rename from fig/csv-mistake.png
rename to episodes/fig/csv-mistake.png
diff --git a/fig/data_validation.png b/episodes/fig/data_validation.png
similarity index 100%
rename from fig/data_validation.png
rename to episodes/fig/data_validation.png
diff --git a/fig/data_validation_window.png b/episodes/fig/data_validation_window.png
similarity index 100%
rename from fig/data_validation_window.png
rename to episodes/fig/data_validation_window.png
diff --git a/fig/drop_down_list.png b/episodes/fig/drop_down_list.png
similarity index 100%
rename from fig/drop_down_list.png
rename to episodes/fig/drop_down_list.png
diff --git a/fig/drop_down_list2.png b/episodes/fig/drop_down_list2.png
similarity index 100%
rename from fig/drop_down_list2.png
rename to episodes/fig/drop_down_list2.png
diff --git a/fig/error_alert.png b/episodes/fig/error_alert.png
similarity index 100%
rename from fig/error_alert.png
rename to episodes/fig/error_alert.png
diff --git a/fig/excel-to-csv.png b/episodes/fig/excel-to-csv.png
similarity index 100%
rename from fig/excel-to-csv.png
rename to episodes/fig/excel-to-csv.png
diff --git a/fig/excel_tables_example.png b/episodes/fig/excel_tables_example.png
similarity index 100%
rename from fig/excel_tables_example.png
rename to episodes/fig/excel_tables_example.png
diff --git a/fig/excel_tables_example1.png b/episodes/fig/excel_tables_example1.png
similarity index 100%
rename from fig/excel_tables_example1.png
rename to episodes/fig/excel_tables_example1.png
diff --git a/fig/formatting.png b/episodes/fig/formatting.png
similarity index 100%
rename from fig/formatting.png
rename to episodes/fig/formatting.png
diff --git a/fig/good_formatting.png b/episodes/fig/good_formatting.png
similarity index 100%
rename from fig/good_formatting.png
rename to episodes/fig/good_formatting.png
diff --git a/fig/input_message.png b/episodes/fig/input_message.png
similarity index 100%
rename from fig/input_message.png
rename to episodes/fig/input_message.png
diff --git a/fig/invalid_value.png b/episodes/fig/invalid_value.png
similarity index 100%
rename from fig/invalid_value.png
rename to episodes/fig/invalid_value.png
diff --git a/fig/multiple-info.png b/episodes/fig/multiple-info.png
similarity index 100%
rename from fig/multiple-info.png
rename to episodes/fig/multiple-info.png
diff --git a/fig/plot_validation.png b/episodes/fig/plot_validation.png
similarity index 100%
rename from fig/plot_validation.png
rename to episodes/fig/plot_validation.png
diff --git a/fig/single-info.png b/episodes/fig/single-info.png
similarity index 100%
rename from fig/single-info.png
rename to episodes/fig/single-info.png
diff --git a/fig/solution_exercise_1_dates.png b/episodes/fig/solution_exercise_1_dates.png
similarity index 100%
rename from fig/solution_exercise_1_dates.png
rename to episodes/fig/solution_exercise_1_dates.png
diff --git a/fig/sorting.png b/episodes/fig/sorting.png
similarity index 100%
rename from fig/sorting.png
rename to episodes/fig/sorting.png
diff --git a/fig/sorting_button.png b/episodes/fig/sorting_button.png
similarity index 100%
rename from fig/sorting_button.png
rename to episodes/fig/sorting_button.png
diff --git a/fig/sorting_solution_1.png b/episodes/fig/sorting_solution_1.png
similarity index 100%
rename from fig/sorting_solution_1.png
rename to episodes/fig/sorting_solution_1.png
diff --git a/fig/sorting_solution_2.png b/episodes/fig/sorting_solution_2.png
similarity index 100%
rename from fig/sorting_solution_2.png
rename to episodes/fig/sorting_solution_2.png
diff --git a/fig/spreadsheet-setup-updated.png b/episodes/fig/spreadsheet-setup-updated.png
similarity index 100%
rename from fig/spreadsheet-setup-updated.png
rename to episodes/fig/spreadsheet-setup-updated.png
diff --git a/fig/spreadsheet-setup.png b/episodes/fig/spreadsheet-setup.png
similarity index 100%
rename from fig/spreadsheet-setup.png
rename to episodes/fig/spreadsheet-setup.png
diff --git a/index.md b/index.md
index 791fe7d..68a8953 100644
--- a/index.md
+++ b/index.md
@@ -1,18 +1,23 @@
---
-layout: lesson
+site: sandpaper::sandpaper_site
---
-{% include base_path.html %}
+> **ATTENTION** This is an experimental test of [The Carpentries Workbench](https://carpentries.github.io/workbench) lesson infrastructure.
+> It was automatically converted from the source lesson via [the lesson transition script](https://github.com/carpentries/lesson-transition/).
+>
+> If anything seems off, please contact Zhian Kamvar [zkamvar@carpentries.org](mailto:zkamvar@carpentries.org)
-Good data organization is the foundation of any research project. Most
+
+
+Good data organization is the foundation of any research project. Most
researchers have data in spreadsheets, so it's the place that many research
-projects start.
+projects start.
-We organize data in spreadsheets in the ways that we as humans want to work with the data,
+We organize data in spreadsheets in the ways that we as humans want to work with the data,
but computers require that data be organized in particular ways. In order
-to use tools that make computation more efficient, such as programming
-languages like R or Python, we need to structure our data the way that
-computers need the data. Since this is where most research projects start,
+to use tools that make computation more efficient, such as programming
+languages like R or Python, we need to structure our data the way that
+computers need the data. Since this is where most research projects start,
this is where we want to start too!
In this lesson, you will learn:
@@ -27,29 +32,39 @@ In this lesson, however, you will *not* learn about data analysis with spreadshe
Much of your time as a researcher will be spent in the initial 'data wrangling'
stage, where you need to organize the data to perform a proper analysis later.
It's not the most fun, but it is necessary. In this lesson you will
-learn how to think about data organization and some practices for more
+learn how to think about data organization and some practices for more
effective data wrangling. With this approach you can better format current data
and plan new data collection so less data wrangling is needed.
+:::::::::::::::::::::::::::::::::::::::::: prereq
+
+## Getting Started
+
+Data Carpentry's teaching is hands-on, so participants are encouraged to use
+their own computers to insure the proper setup of tools for an efficient
+workflow. **These lessons assume no prior knowledge of the skills or tools.** + +To get started, follow the directions in the "[Setup](learners/setup.md)" tab to +download data to your computer and follow any installation instructions. + +#### Prerequisites + +This lesson requires a working copy of spreadsheet software, such as Microsoft +Excel or LibreOffice or OpenOffice.org (see more details in "[Setup](learners/setup.md)"). +
To most effectively use these materials, please make sure to install +everything *before* working through this lesson. + + +:::::::::::::::::::::::::::::::::::::::::::::::::: + +:::::::::::::::::::::::::::::::::::::::::: prereq + +## For Instructors + +If you are teaching this lesson in a workshop, please see the +[Instructor notes](instructors/instructor-notes.md). + + +:::::::::::::::::::::::::::::::::::::::::::::::::: + -> ## Getting Started -> -> Data Carpentry's teaching is hands-on, so participants are encouraged to use -> their own computers to insure the proper setup of tools for an efficient -> workflow.
**These lessons assume no prior knowledge of the skills or tools.** -> -> To get started, follow the directions in the "[Setup]({{ relative_root_path }}/{% link setup.md %})" tab to -> download data to your computer and follow any installation instructions. -> -> #### Prerequisites -> -> This lesson requires a working copy of spreadsheet software, such as Microsoft -> Excel or LibreOffice or OpenOffice.org (see more details in "[Setup]({{ relative_root_path }}/{% link setup.md %})"). ->
To most effectively use these materials, please make sure to install -> everything *before* working through this lesson. -{: .prereq} - -> ## For Instructors -> If you are teaching this lesson in a workshop, please see the -> [Instructor notes](guide/). -{: .prereq} diff --git a/_extras/datamanagement.md b/instructors/datamanagement.md similarity index 65% rename from _extras/datamanagement.md rename to instructors/datamanagement.md index 3df95ef..01400ca 100644 --- a/_extras/datamanagement.md +++ b/instructors/datamanagement.md @@ -1,31 +1,33 @@ --- -layout: page title: Data Management Tips --- -[The following is distilled from a _Nature_ Toolbox article that was published on 1 April 2019: Perkel, J.M. 11 ways to avert a data-storage disaster. _Nature_ **568**, 131-132 (2019). [https://www.nature.com/articles/d41586-019-01040-w](https://www.nature.com/articles/d41586-019-01040-w)] + +[The following is distilled from a *Nature* Toolbox article that was published on 1 April 2019: Perkel, J.M. 11 ways to avert a data-storage disaster. *Nature* **568**, 131-132 (2019). [https://www.nature.com/articles/d41586-019-01040-w](https://www.nature.com/articles/d41586-019-01040-w)] **Tips for managing your data** 1. **3-2-1**: Fires, floods, and theft all happen -- not to mention bitrot (the deterioration of storage media over time). So, keep at least three copies of your data, on two different media, at least one of which is off-site. (And be sure to store your local copies under proper environmental conditions!) -2. **Talk to the experts**: Make your institution’s professionals your first call. Ask IT team colleagues about free or low-cost backup options available; your librarian about data management strategies; and your grant officers about regulations regarding how, and how long, to store data +2. **Talk to the experts**: Make your institution's professionals your first call. Ask IT team colleagues about free or low-cost backup options available; your librarian about data management strategies; and your grant officers about regulations regarding how, and how long, to store data -3. **Safeguard privacy**: Private data, such as student information, cannot be stored just anywhere. And if those data are lost or stolen, you could face consequences. If you have, or plan to store such data, speak to your institution’s IT team for advice. +3. **Safeguard privacy**: Private data, such as student information, cannot be stored just anywhere. And if those data are lost or stolen, you could face consequences. If you have, or plan to store such data, speak to your institution's IT team for advice. -4. **Manage your data**: Make your backups more effective, and more future-proof, by developing a data-management plan. Establish file-naming conventions and organizational strategies -- for instance, that each project gets its own directory, with dedicated subdirectories for data and code. Decide which data will be backed up, and which can be discarded. Determine where different data types will be backed up, and how often. And, document everything: Keep a copy of your data management plan where people can find it, and annotate your experiments with README files that indicate the experiment, the file structures, required applications or scripts, and so on. +4. **Manage your data**: Make your backups more effective, and more future-proof, by developing a data-management plan. Establish file-naming conventions and organizational strategies -- for instance, that each project gets its own directory, with dedicated subdirectories for data and code. Decide which data will be backed up, and which can be discarded. Determine where different data types will be backed up, and how often. And, document everything: Keep a copy of your data management plan where people can find it, and annotate your experiments with README files that indicate the experiment, the file structures, required applications or scripts, and so on. -5. **Share with others**: Your colleagues may also need access to your data. Can someone else understand what the data are and where they’re located if you’re not around? Make sure they have access and permissions to the data and that they are able to understand what the files are and how they are organized (see **Manage your data**). +5. **Share with others**: Your colleagues may also need access to your data. Can someone else understand what the data are and where they're located if you're not around? Make sure they have access and permissions to the data and that they are able to understand what the files are and how they are organized (see **Manage your data**). 6. **Be realistic**: Once you develop a backup strategy, discuss it in the lab. Is it accessible to new colleagues, or only command-line experts? Is it doable after pulling an all-nighter? And how effective is it? Simulate what would happen if disaster should strike: What data will you lose, and what can you recover? -7. **Automate**: The thing about backups is, you need them when you least expect it. So don’t rely on remembering to run a backup -- automate them. +7. **Automate**: The thing about backups is, you need them when you least expect it. So don't rely on remembering to run a backup -- automate them. -8. **Test your backup**: Do your backups actually work? Test them to find out. Make sure you can actually open key files, and that you have the required applications to read them. Use checksums to ensure data integrity. And test the files on a different system if possible, as, if your primary computer should fail, you won’t have access to its contents. +8. **Test your backup**: Do your backups actually work? Test them to find out. Make sure you can actually open key files, and that you have the required applications to read them. Use checksums to ensure data integrity. And test the files on a different system if possible, as, if your primary computer should fail, you won't have access to its contents. -9. **Protect your raw data**: Always backup your raw data. And keep it safe: Duplicate the data before working with it, and open it read-only. An errant file-open command in write mode (e.g., in Python: `f = open (filename, ‘w’)`) instead of read (`‘r’`) is all it takes to wipe out a file for good. +9. **Protect your raw data**: Always backup your raw data. And keep it safe: Duplicate the data before working with it, and open it read-only. An errant file-open command in write mode (e.g., in Python: `f = open (filename, ‘w')`) instead of read (`‘r'`) is all it takes to wipe out a file for good. 10. **Keep one backup offline**: An always-on backup system is a recipe for disaster. If your computer is hacked, or suffers a power-spike, the backup system can also be compromised. So keep at least one backup offline, just in case. -11. **Plan ahead**: Storage media don’t last forever, so periodically review your backup strategy to make sure it’s current. Do you still have devices that can read the backups? Is it time to migrate to a newer platform? And don’t neglect your cloud storage: companies can shift priorities, or you can lose your passwords. So double-up on those too, just in case. +11. **Plan ahead**: Storage media don't last forever, so periodically review your backup strategy to make sure it's current. Do you still have devices that can read the backups? Is it time to migrate to a newer platform? And don't neglect your cloud storage: companies can shift priorities, or you can lose your passwords. So double-up on those too, just in case. + +12. **Expect the unexpected**: Make periodic disaster assessments to safeguard your hardware. If you live in a flood zone, the basement may not be the best place to store a server. If you live in an earthquake-prone area, you might anchor your computers so they don't fall over. And if your computers are located near fire-control sprinklers, you might raise them off the ground, or shield them from potential leaks. + -12. **Expect the unexpected**: Make periodic disaster assessments to safeguard your hardware. If you live in a flood zone, the basement may not be the best place to store a server. If you live in an earthquake-prone area, you might anchor your computers so they don’t fall over. And if your computers are located near fire-control sprinklers, you might raise them off the ground, or shield them from potential leaks. diff --git a/instructors/instructor-notes.md b/instructors/instructor-notes.md new file mode 100644 index 0000000..c596b42 --- /dev/null +++ b/instructors/instructor-notes.md @@ -0,0 +1,115 @@ +--- +title: Instructor Notes +--- + +## Instructor notes + +## Lesson motivation and learning objectives + +The purpose of this lesson is *not* to teach how to do data analysis in spreadsheets, +but to teach good data organization and how to do some data cleaning and +quality control in a spreadsheet program. + +## Lesson design + +#### [Introduction](../episodes/00-intro.md) + +- Introduce that we're teaching data organization, and that we're using + spreadsheets, because most people do data entry in spreadsheets or + have data in spreadsheets. +- Emphasize that we are teaching good practice in data organization and that + this is the foundation of their research practice. Without organized and clean + data, it will be difficult for them to apply the things we're teaching in the + rest of the workshop to their data. +- Much of their lives as a researcher will be spent on this 'data wrangling' stage, but + some of it can be prevented with good strategies for data collection up front. +- Tell that we're not teaching data analysis or plotting in spreadsheets, because it's + very manual and also not reproducible. That's why we're teaching SQL, R, Python! +- Now let's talk about spreadsheets, and when we say spreadsheets, we mean any program that + does spreadsheets like Excel, LibreOffice, OpenOffice. Most learners are probably using Excel. +- Ask the audience any things they've accidentally done in spreadsheets. Talk about an example of your own, like that you accidentally sorted only a single column and not the rest + of the data in the spreadsheet. What are the pain points!? +- As people answer highlight some of these issues with spreadsheets + +#### [Formatting data](../episodes/01-format-data.md) + +- Go through the point about keeping track of your steps and keeping raw data raw +- Go through the cardinal rule of spreadsheets about columns, rows and cells +- Hand them a messy data file and have them pair up and work together to clean up the data. + *Give them 15 minutes to do this.* +- Ask for what people did to clean the data. As they bring up different points you can + refer to them in the 02-common-mistakes.md file, or expand a bit on the point they brought up. + If you are just teaching the lesson, it would be good to familiarize yourself with + the set of mistakes in 02-common-mistakes. All these mistakes are present in the messy + dataset. +- If you get a response where they've fixed the date, you can pause and go to the + 03-dates-as-data.md lesson. Or you can say you'll come back to dates at the end. + There's an exercise in that file about how to change the + date into three columns using Excel's built in MONTH, DAY, YEAR functions. Have them + run through that exercise. + +#### [Common formatting problems](../episodes/02-common-mistakes.md) + +- **Don't go through this chapter** except to refer to as responses to the exercise in + the previous chapter. + +#### [Dates as data](../episodes/03-dates-as-data.md) + +- Do the exercise and make the point about dates either in response to a learner bringing + up date as an issue during the responses, or at the end of the response time. + +#### [Quality control](../episodes/04-quality-control.md) + +*This lesson is optional* + +The challenge with this lesson is that the instructor's version of the spreadsheet software is going to look different than about half the room's. It makes +it challenging to show where you can find menu options and navigate through. + +Instead discuss the concepts of quality control, and how things like sorting can help you find outliers in your data. + +#### [Exporting data](../episodes/05-exporting-data.md) + +- Have the students export their cleaned data as CSV. Reiterate again the need for + data in this format for the other tools we'll be using. + +#### Concluding points + +- Now your data is organized so that a computer can read and understand it. This + let's you use the full power of the computer for your analyses as we'll see in the + rest of the workshop. +- While your data is now neatly organized, it still might have errors or missing data + or other problems. It's like you put all your data in the right drawers, but the + drawers might still be messy. The next lesson is going to teach you OpenRefine which + is great for data cleaning and for some of the quality control that we touched on + in this lesson. It also has the advantage that it automatically keeps track of the + steps you take. + +## Technical tips and tricks + +Provide information on setting up your environment for learners to view your +live coding (increasing text size, changing text color, etc), as well as +general recommendations for working with coding tools to best suit the +learning environment. + +## Common problems + +#### Excel looks and acts different on different operating systems + +The main challenge with this lesson is that Excel looks very different and how you +do things is even different between Mac and PC, and between different versions of +Excel. So, the presenter's environment will only be the same as some of the learners. + +We need better notes and screenshots of how things work on both Mac and PC. But we +likely won't be able to cover all the different versions of Excel. + +If you have a helper who has experience with the other OS than you, it would be good +to prep them to help with this lesson and tell how people to do things in the other OS. + +#### People are not interactive or responsive on the Exercise + +This lesson depends on people working on the exercise and responding with things +that are fixed. If your audience is reluctant to participate, start out with +some things on your own, or ask a helper for their answers. This generally gets +even a reluctant audience started. + + diff --git a/_extras/discuss.md b/learners/discuss.md similarity index 76% rename from _extras/discuss.md rename to learners/discuss.md index b169e94..983d483 100644 --- a/_extras/discuss.md +++ b/learners/discuss.md @@ -1,36 +1,39 @@ --- -layout: page title: Discussion --- ### Dealing with commas as part of data values in `csv` files -When talking about [exporting data](../05-exporting-data/) we discussed how to export Excel file formats into `csv`. Comma Separated Value files are indeed very useful and allow for easily exchanging and sharing data. +When talking about [exporting data](../episodes/05-exporting-data.md) we discussed how to export Excel file formats into `csv`. Comma Separated Value files are indeed very useful and allow for easily exchanging and sharing data. However, there are some significant problems with this particular format. Quite often the data values themselves may include commas (,). In that case, the software which you use (including Excel) will most likely incorrectly display the data in columns. This is because the commas which are a part of the data values will be interpreted as delimiters. For example, our data might look like this: - - species_id,genus,species,taxa - AB,Amphispiza,bilineata,Bird - AH,Ammospermophilus,harrisi,Rodent-not,censused - AS,Ammodramus,savannarum,Bird - BA,Baiomys,taylori,Rodent - -In the record `AH,Ammospermophilus,harrisi,Rodent-not,censused` the value for `taxa` includes a comma (`Rodent-not,censused`). + +``` + species_id,genus,species,taxa + AB,Amphispiza,bilineata,Bird + AH,Ammospermophilus,harrisi,Rodent-not,censused + AS,Ammodramus,savannarum,Bird + BA,Baiomys,taylori,Rodent +``` + +In the record `AH,Ammospermophilus,harrisi,Rodent-not,censused` the value for `taxa` includes a comma (`Rodent-not,censused`). If we try to read the above into Excel (or other spreadsheet program), we will get something like this: - +{alt='Issue with importing csv format'} -The value for `taxa` was split into two columns (instead of being put in one column `D`). This can propagate to a number of further errors. For example, the extra column will be interpreted as a column with many missing values (and without a proper header). In addition to that, the value in column `D` for the record in row 3 (so the one where the value for 'taxa' contained the comma) is now incorrect. +The value for `taxa` was split into two columns (instead of being put in one column `D`). This can propagate to a number of further errors. For example, the extra column will be interpreted as a column with many missing values (and without a proper header). In addition to that, the value in column `D` for the record in row 3 (so the one where the value for 'taxa' contained the comma) is now incorrect. If you want to store your data in `csv` format and expect that your data values may contain commas, you can avoid the problem discussed above by putting the values in quotes (""). Applying this rule, our data might look like this: - species_id,genus,species,taxa - "AB","Amphispiza","bilineata","Bird" - "AH","Ammospermophilus","harrisi","Rodent-not, censused" - "AS","Ammodramus","savannarum","Bird" - "BA","Baiomys","taylori","Rodent" +``` +species_id,genus,species,taxa +"AB","Amphispiza","bilineata","Bird" +"AH","Ammospermophilus","harrisi","Rodent-not, censused" +"AS","Ammodramus","savannarum","Bird" +"BA","Baiomys","taylori","Rodent" +``` Now opening this file as a `csv` in Excel will not lead to an extra column, because Excel will only use commas that fall outside of quotation marks as delimiting characters. However, if you are working with an already existing dataset in which the data values are not included in "" but which have commas as both delimiters and parts of data values, you are potentially facing a major problem with data cleaning. @@ -38,3 +41,4 @@ If the dataset you're dealing with contains hundreds or thousands of records, cl Cleaning up datasets is one of the major problems in many scientific disciplines. The approach almost always depends on the particular context. However, it is a good practice to clean the data in an automated fashion, for example by writing and running a script. The Python and R lessons will give you the basis for developing skills to build relevant scripts. + diff --git a/reference.md b/learners/reference.md similarity index 88% rename from reference.md rename to learners/reference.md index 2377355..9973df6 100644 --- a/reference.md +++ b/learners/reference.md @@ -1,15 +1,15 @@ --- -layout: reference +title: 'Glossary' --- ## Glossary -{:auto_ids} +{:auto\_ids} cleaned data : data that has been manipulated post-collection to remove errors or inaccuracies, introduce desired formatting changes, or otherwise prepare the data for analysis conditional formatting -: formatting that is applied to a specific cell or range of cells depending on a set of criteria +: formatting that is applied to a specific cell or range of cells depending on a set of criteria CSV (comma separated values) format : a plain text file format in which values are separated by commas @@ -18,7 +18,7 @@ factor : a variable that takes on a limited number of possible values (i.e. categorical data) metadata -: data which describes other data +: data which describes other data null value : a value used to record observations missing from a dataset @@ -30,7 +30,7 @@ plain text : unformatted text quality assurance -: any process which checks data for validity during entry +: any process which checks data for validity during entry quality control : any process which removes problematic data from a dataset @@ -49,3 +49,5 @@ TSV (tab separated values) format variable : a category of data being collected on the object being recorded (e.g. a mouse's weight) + + diff --git a/learners/setup.md b/learners/setup.md new file mode 100644 index 0000000..2502d2c --- /dev/null +++ b/learners/setup.md @@ -0,0 +1,58 @@ +--- +title: Setup +--- + +:::::::::::::::::::::::::::::::::::::::::: prereq + +## Data + +**Download** this data file to your computer: [https://ndownloader.figshare.com/files/2252083](https://ndownloader.figshare.com/files/2252083) + +#### About the data + +The data for this lesson is a part of the Data Carpentry Ecology workshop. +It is a teaching version of the Portal Database. The data in this lesson +is a subset of the teaching version that has been intentionally 'messed up' +for this lesson. + +The data for this lesson and the workshop are in the +[Portal Project Teaching Database](https://figshare.com/articles/Portal_Project_Teaching_Database/1314459) +available on FigShare, with a CC-BY license +available for reuse. + +> Ernest, M., Brown, J., Valone, T., and White, E.P. (2017). Portal Project Teaching Database. Version 6. Figshare. [DOI: 10.6084/m9.figshare.1314459.v6](https://figshare.com/articles/Portal_Project_Teaching_Database/1314459) + +:::::::::::::::::::::::::::::::::::::::::::::::::: + +:::::::::::::::::::::::::::::::::::::::::: prereq + +## Software + +To interact with spreadsheets, we can use LibreOffice, Microsoft Excel, Gnumeric, OpenOffice.org, or other programs. Commands may differ a bit between programs, but the general ideas for thinking about spreadsheets are the same. + +For this lesson, if you don't have a spreadsheet program already, you can use LibreOffice. It's a free, open source spreadsheet program. + +#### Windows + +- Download the Installer + - Install LibreOffice by going to [the installation page](https://www.libreoffice.org/download/libreoffice-fresh/). The version for Windows should automatically be selected. Click Download Version X.X.X (whichever is the most recent version). You will go to a page that asks about a donation, but you don't need to make one. Your download should begin automatically. +- Install LibreOffice +- Once the installer is downloaded, double click on it and LibreOffice should install. + +#### Mac OS X + +- Download the Installer + - Install LibreOffice by going to [the installation page](https://www.libreoffice.org/download/libreoffice-fresh/). The version for Mac should automatically be selected. Click Download Version X.X.X (whichever is the most recent version). You will go to a page that asks about a donation, but you don't need to make one. Your download should begin automatically. +- Install LibreOffice +- Once the installer is downloaded, double click on it and LibreOffice should install. + +#### Linux + +- Download the Installer + - Install LibreOffice by going to [the installation page](https://www.libreoffice.org/download/libreoffice-fresh/). The version for Linux should automatically be selected. Click Download Version X.X.X (whichever is the most recent version). You will go to a page that asks about a donation, but you don't need to make one. Your download should begin automatically. +- Install LibreOffice +- Once the installer is downloaded, double click on it and LibreOffice should install. + +:::::::::::::::::::::::::::::::::::::::::::::::::: + + diff --git a/profiles/learner-profiles.md b/profiles/learner-profiles.md new file mode 100644 index 0000000..434e335 --- /dev/null +++ b/profiles/learner-profiles.md @@ -0,0 +1,5 @@ +--- +title: FIXME +--- + +This is a placeholder file. Please add content here. diff --git a/setup.md b/setup.md deleted file mode 100644 index 89b4026..0000000 --- a/setup.md +++ /dev/null @@ -1,53 +0,0 @@ ---- -layout: page -title: Setup ---- - -> ## Data -> -> **Download** this data file to your computer: [https://ndownloader.figshare.com/files/2252083](https://ndownloader.figshare.com/files/2252083) -> -> #### About the data -> The data for this lesson is a part of the Data Carpentry Ecology workshop. -> It is a teaching version of the Portal Database. The data in this lesson -> is a subset of the teaching version that has been intentionally 'messed up' -> for this lesson. -> -> The data for this lesson and the workshop are in the -> [Portal Project Teaching Database](https://figshare.com/articles/Portal_Project_Teaching_Database/1314459) -> available on FigShare, with a CC-BY license -> available for reuse. -> -> > Ernest, M., Brown, J., Valone, T., and White, E.P. (2017). Portal Project Teaching Database. Version 6. Figshare. [DOI: 10.6084/m9.figshare.1314459.v6](https://figshare.com/articles/Portal_Project_Teaching_Database/1314459) -> -{: .prereq} - -> ## Software -> -> To interact with spreadsheets, we can use LibreOffice, Microsoft Excel, Gnumeric, OpenOffice.org, or other programs. Commands may differ a bit between programs, but the general ideas for thinking about spreadsheets are the same. -> -> For this lesson, if you don't have a spreadsheet program already, you can use LibreOffice. It's a free, open source spreadsheet program. -> -> #### Windows -> -> - Download the Installer -> - Install LibreOffice by going to [the installation page](https://www.libreoffice.org/download/libreoffice-fresh/). The version for Windows should automatically be selected. Click Download Version X.X.X (whichever is the most recent version). You will go to a page that asks about a donation, but you don't need to make one. Your download should begin automatically. -> - Install LibreOffice -> - Once the installer is downloaded, double click on it and LibreOffice should install. -> -> #### Mac OS X -> -> - Download the Installer -> - Install LibreOffice by going to [the installation page](https://www.libreoffice.org/download/libreoffice-fresh/). The version for Mac should automatically be selected. Click Download Version X.X.X (whichever is the most recent version). You will go to a page that asks about a donation, but you don't need to make one. Your download should begin automatically. -> - Install LibreOffice -> - Once the installer is downloaded, double click on it and LibreOffice should install. -> -> -> #### Linux -> -> - Download the Installer -> - Install LibreOffice by going to [the installation page](https://www.libreoffice.org/download/libreoffice-fresh/). The version for Linux should automatically be selected. Click Download Version X.X.X (whichever is the most recent version). You will go to a page that asks about a donation, but you don't need to make one. Your download should begin automatically. -> - Install LibreOffice -> - Once the installer is downloaded, double click on it and LibreOffice should install. -> -{: .prereq} diff --git a/site/README.md b/site/README.md new file mode 100644 index 0000000..42997e3 --- /dev/null +++ b/site/README.md @@ -0,0 +1,2 @@ +This directory contains rendered lesson materials. Please do not edit files +here.