EDA .Rmd

---
title: "EDA S&P 500 index"
date: "`r Sys.Date()`"
output:
  html_document:
    code_folding: hide
    number_sections: false
    toc: yes
    toc_depth: 3
    toc_float: yes
  pdf_document:
    toc: yes
    toc_depth: '3'
---

```{r init, include=FALSE}
# Load all required packagess
library(xts)
library(ezids)
library(dplyr)
library(tseries)
library(ggplot2)
library(forecast)
library(corrplot)
library(lubridate)

knitr::opts_chunk$set(warning = F, results = "markup", message = F)
options(scientific=T, digits = 3) 
```


```{r}
df <- data.frame(read.csv('sp500.csv'))
df$Date <- as.Date(df$Date)
str(df)
```


# finding unique values in Symbols column
```{r}
length(df$Symbol)
unique(df$GICS )
unique(df$Symbol)

```


# count plot for different sectors
```{r}
library(ggplot2)
ggplot(df, aes(x = GICS)) + geom_bar() + theme(axis.text.x = element_text(angle = 90, hjust = 1))

```
In the overall dataset, the sector with the highest number of symbols is the Industrials sector, followed by the Financials sector. the least number of symbols is in the communication Services sector.



# sort dataframe by date column
```{r}
df <- df[order(df$Date),]
head(df)

```


# finding unique values in the Date column and get year from date column
```{r}
library(lubridate)
year <- unique(year(df$Date))
year
```


# plot countplot for each year
```{r}
ggplot(df, aes(x = year(df$Date))) + geom_bar() + theme(axis.text.x = element_text(angle = 90, hjust = 1))

```
the number of symbols in the dataset has increased over the years. The year with the highest number of symbols is 2020, followed by 2021. The year with the least number of symbols is 2000. there was a significant increase in companies because of the fluctuations in the stock market.


# nrows where the year is 2020
```{r}
(nrow(df[year(df$Date) == 2020,]))
unique(df$Symbol[year(df$Date) == 2023])

```

#plot countplot for each year and sector
```{r}
ggplot(df, aes(x = year(df$Date), fill = GICS)) + geom_bar() + theme(axis.text.x = element_text(angle = 90, hjust = 1))

```
the plot tells us the contribution of each sector to the total number of symbols in the dataset. 


# plot sum of volume for each year
```{r}
ggplot(df, aes(x = year(df$Date), y = Volume)) + geom_bar(stat = "identity") + theme(axis.text.x = element_text(angle = 90, hjust = 1))
```
the plot tells us the total volume of stocks traded in each year. The year with the highest volume of stocks traded is 2008. The year with the least volume of stocks traded is 2000.


# sum of volume for each year
```{r}
df_sum <- df %>% group_by(year(df$Date)) %>% summarise(sum(Volume))
df_sum
```
volume of stocks traded in numeric values for each year.


# plot sum of volume for each year and sector
```{r}
ggplot(df, aes(x = year(df$Date), y = Volume, fill = GICS)) + geom_bar(stat = "identity") + theme(axis.text.x = element_text(angle = 90, hjust = 1))

```
the plot tells us the contribution of each sector to the total volume of stocks traded in the dataset.




# sum of volume for each year and sector
```{r}
library(dplyr)
library(lubridate)
df_sum_grouped <- df %>% group_by(year(df$Date), GICS) %>% summarise(sum(Volume))
(df_sum_grouped)
```
the contribution of each sector to the total volume of stocks traded in the dataset in numeric values.


# sort dataframe by close column
```{r}
df_ <- df[order(df$Close),]
head(df_)
```



# plot highest closing price for each year for each sector
```{r}
ggplot(df, aes(x = year(df$Date), y = Close, color = GICS)) + geom_point() + theme(axis.text.x = element_text(angle = 90, hjust = 1))
```

the plot tells us the highest closing price for each sector in each year.



# top ten symbols with highest close price each year
```{r}
top_ten <- df %>% group_by(year(df$Date)) %>% top_n(10, Close)
top_ten
```