README.Rmd

---
output: github_document
always_allow_html: true
editor_options: 
  markdown: 
    wrap: 72
  chunk_output_type: console
---

<!-- README.md is generated from README.Rmd. Please edit that file -->

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%",
  message = FALSE,
  warning = FALSE,
  fig.retina = 2,
  fig.align = 'center'
)

library(tidyverse)
library(wordcloud2)
```

# washopenresearch

<!-- badges: start -->

[![License: CC BY
4.0](https://img.shields.io/badge/License-CC_BY_4.0-lightgrey.svg)](https://creativecommons.org/licenses/by/4.0/)
[![R-CMD-check](https://github.com/openwashdata/washopenresearch/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/openwashdata/washopenresearch/actions/workflows/R-CMD-check.yaml)
[![DOI](https://zenodo.org/badge/734215029.svg)](https://zenodo.org/doi/10.5281/zenodo.11185699)

<!-- badges: end -->

The goal of washopenresearch is to provide an overview of open research
data related to Water Sanitation and Hygiene (WASH). The current version
contains two datasets from the following sources:

-   `washdev`: Open access journal [*Journal of Water, Sanitation and
    Hygiene for Development*](https://iwaponline.com/washdev)
-   `uncnewsletter`: Research section of the newsletter [North Carolina
    Water
    News](https://waterinstitute.unc.edu/our-work/nc-water-news-newsletter)

![](man/figures/washdev_wordcloud.png){width="515"}

## Installation

You can install the development version of washopenresearch from
[GitHub](https://github.com/) with:

``` r
# install.packages("devtools")
devtools::install_github("openwashdata/washopenresearch")
```

Alternatively, you can download the individual datasets as a CSV or XLSX
file from the table below.

```{r, echo=FALSE, message=FALSE, warning=FALSE}

extdata_path <- "https://github.com/openwashdata/washopenresearch/raw/main/inst/extdata/"

read_csv("data-raw/dictionary.csv") |> 
  distinct(file_name) |> 
  dplyr::mutate(file_name = str_remove(file_name, ".rda")) |> 
  dplyr::rename(dataset = file_name) |> 
  mutate(
    CSV = paste0("[Download CSV](", extdata_path, dataset, ".csv)"),
    XLSX = paste0("[Download XLSX](", extdata_path, dataset, ".xlsx)")
  ) |> 
  knitr::kable()

```

## Data

The package provides access to two datasets `washdev` and
`uncnewsletter`. Each dataset collects information on scientific
articles about (1) article metadata (e.g. title, first author,
correspondence author), (2) supplementary material information, (3) data
availability statement, and (4) semantic information (e.g. keywords).

```{r}
library(washopenresearch)
```

### washdev

The dataset `washdev` contains data on open access articles of the
*Journal of Water, Sanitation & Hygiene for Development* (Vol.1 Issue
1 - Vol.13 Issue 11). It has `r nrow(washdev)` observations from March
2011 to November 2023.

```{r}
washdev |> 
  head(3) |> 
  gt::gt() |>
  gt::as_raw_html()
```

For an overview of the variable names, see the following table.

```{r echo=FALSE, message=FALSE, warning=FALSE}
readr::read_csv("data-raw/dictionary.csv") |>
  dplyr::filter(file_name == "washdev.rda") |>
  dplyr::select(variable_name:description) |> 
  knitr::kable() |> 
  kableExtra::kable_styling("striped") |> 
  kableExtra::scroll_box(height = "200px")
```

### uncnewsletter

The dataset `uncnewsletter` contains data on a curated list of articles
published at the Research section of the newsletter North Carolina Water
News. It has `r nrow(uncnewsletter)` observations from 2020 to 2023.

```{r}
uncnewsletter |> 
  head(3) |> 
  gt::gt() |>
  gt::as_raw_html()
```

For an overview of the variable descriptions, see the following table.

```{r echo=FALSE, message=FALSE, warning=FALSE}
readr::read_csv("data-raw/dictionary.csv") |>
  dplyr::filter(file_name == "uncnewsletter.rda") |>
  dplyr::select(variable_name:description) |> 
  knitr::kable() |> 
  kableExtra::kable_styling("striped") |> 
  kableExtra::scroll_box(height = "200px")
```


## Example

### washdev

1.  What are the top 10 countries(or regions) the first authors from in
    the *Journal of Water, Sanitation and Hygiene for Development*?

```{r}
library(washopenresearch)

washdev |> 
  filter(!is.na(first_author_affiliation_country)) |>
  group_by(first_author_affiliation_country) |>
  summarise(count=n()) |>
  arrange(desc(count)) |>
  head(10) |>
  ggplot() +
    geom_col(aes(x = reorder(first_author_affiliation_country, count), 
                 y = count)) +
    labs(title = "Top 10 countries of first author",
        subtitle = "in the Journal of Water, Sanitation and Hygiene for Development",
        x = "First Author Country", y = "Count") +
    scale_x_discrete(labels = scales::label_wrap(15))+
    coord_flip() +
    theme_classic()
```

2.  What are the top choices of keywords in WASH Dev?

Each publication may provide a list of keywords, typically 5-7, to
summarize the topics of the article. Here we compile all keywords and
calculate their frequency to be used.

```{r washdev_keyword_frequency, echo=TRUE}
keywords_freq <- washdev$keywords |>
    unlist() |>
    str_to_lower() |>
  table() |>
  as.data.frame() |>
  as_tibble() |>
  arrange(desc(Freq))

# Top 20 keywords
ggplot(data = head(keywords_freq, 20)) +
  geom_bar(aes(x = reorder(Var1, Freq), y=Freq), stat = "identity") +
  coord_flip() +
  labs(title = "Top 20 Keywords in WASH Dev Journal", x = "Keywords", y = "Count") +
  theme_bw()
```

```{r washdev_wordcloud, echo=FALSE, eval=FALSE}

keywords_freq <- keywords_freq |>
  rename(word=Var1, freq=Freq) |>
  as.data.frame()

wc <- wordcloud2(keywords_freq)
library("webshot")
library("htmlwidgets")
# webshot::install_phantomjs()
saveWidget(wc, "man/figures/wc.html", selfcontained = F)
webshot("man/figures/wc.html", "man/figures/washdev_wordcloud.png", delay = 3)
```

### uncnewsletter

1.  What are the top 10 source websites of the publications selected by
    the newsletter?

```{r}
uncnewsletter |> 
  group_by(url_source) |>
  summarise(count=n()) |>
  arrange(desc(count)) |>
  head(10) |>
  ggplot() +
    geom_col(aes(x = reorder(url_source, count), 
                 y = count)) +
   labs(title = "Top 10 publication websites",
        subtitle = "in the selection of North Carolina Water News",
        x = "Website URL", y = "Count") +
   scale_x_discrete(labels = scales::label_wrap(15))+
   coord_flip() +
   theme_classic()
```

## Method

We describe the raw data collection procedure of each dataset in this
section. To reproduce the collection, you need to have python3 installed
and install python libraries

```         
pip install requirements.txt
```

### washdev

The collection of `washdev` is via web scraping using Python. The script
can be found in `inst/python/washdev_scraping.py`. First, each
publication link is scraped from iterating the table of contents of all
volumes. This step delivers a table containing the variables paper ID,
volume number, issue number, publication url, journal title, publication
title, and published year. This table will be merged to get the final
dataset.

Then, for each publication, we retrieve the needed variables from the
publication's html file using the publication url. The retrieval is
rule-based to find the relevant fields (e.g. supplementary materials)
and extract the value.

### uncnewsletter

The collection of `uncnewsletter` is a combination of web scraping and
manual annotation. We first use the newsletter archive to scrape all
publication website links. The code can be found at
`inst/python/uncnewsletter_scraping.py`. Two annotators worked on the
manual extraction of the needed variables on these publications. For
each publication, an annotator follows the guide to fill in the value on
an collaborative spreadsheet. The guide is converted into the data
dictionary for this dataset.

## License

Data are available as
[CC-BY](https://github.com/openwashdata/washopenresearch/blob/main/LICENSE.md).

## Citation

Please cite this package using:

```{r}
citation("washopenresearch")
```