cyber-network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to the Dataset\n",
    "\n",
    "Valeriano, B., & Maness, R. C. (2014). The dynamics of cyber conflict between rival antagonists, 2001–11. Journal of Peace Research, 51(3), 347–360. http://www.jstor.org/stable/24557484\n",
    "\n",
    "Each pair of states engaged in cyber conflict has two states involved.\n",
    "Dataset: https://drryanmaness.wixsite.com/cyberconflict/cyber-conflict-dataset\n",
    "\n",
    "- Dyadic Cyber Incident Dataset (DCID)\n",
    "- States: A, B\n",
    "- Cyber incidents: A (initiator) -> B (target)\n",
    "- Dyadic dataset: A <-> B\n",
    "\n",
    "For individual cyber conflicts, the dataset uses the concept of ‘cyber incident.’ Incidents may include thousands of events, but accounting for every single intrusion or attack made is impossible. In this case, the issue is how to count campaigns such as the SolarWinds intrusion? Should the Russian intrusion into SolarWinds be coded as a single campaign, an active intrusion into 110 organizations or a latent intrusion into 10,000? In the CDID data set, SolarWinds is counted as one incident.\n",
    "\n",
    "In addition, it is difficult to identify the perpetrator (attribution) in cyberspace. Therefore we also have to add a caveat to the initiator and target for each incident in the dataset.\n",
    "\n",
    "Codebook: https://a678132e-4067-4ed4-800a-239c80659fd1.filesusr.com/ugd/4b99a4_ca35bdb6bd55443e890d2dab86910b4c.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Libraries Used\n",
    "\n",
    "- pandas\n",
    "- matplotlib\n",
    "- networkx\n",
    "- pyvis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "file_path = 'data/DCID_2.0_Release_update_February_2023.xlsx'\n",
    "# Read file from excel\n",
    "dcid = pd.read_excel(file_path, \"DCID_2.0_Release_update_Februar\")\n",
    "\n",
    "# Avoid duplicates on the Cyberincidentnum \n",
    "# Caution: THIS IS kind of STRANGE but if have to do it if working with this sheet from Excel\n",
    "dcid = dcid.drop_duplicates(subset='Cyberincidentnum', keep='first')\n",
    "\n",
    "print(dcid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the correlates of war (COW) country codes in to dcid to ISO codes\n",
    "cow_countries = pd.read_csv('data/COW-country-codes.csv')\n",
    "print(cow_countries)\n",
    "\n",
    "# check for duplicates in correlates of war country codes\n",
    "# AAARRHHH What!!! there are duplicates in the correlates of war country codes. Why??\n",
    "cow_countries = cow_countries.drop_duplicates(subset='CCode', keep='first')\n",
    "print(cow_countries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dcid = dcid.merge(cow_countries, left_on='initiator', right_on='CCode', how='left', suffixes=('', '_y'))\n",
    "\n",
    "# Replace the target column with a country code from a correlates of war country codes\n",
    "dcid = dcid.merge(cow_countries, left_on='target', right_on='CCode', how='left', suffixes=('', '_y'))\n",
    "\n",
    "# Rename the columns StateAbb to initiator_iso\n",
    "dcid = dcid.rename(columns={'StateAbb': 'initiator_iso'})\n",
    "dcid = dcid.rename(columns={'StateAbb_y': 'target_iso'})\n",
    "\n",
    "print(len(dcid))\n",
    "print(dcid)\n",
    "# save the file\n",
    "# dcid.to_csv('export.csv', index=False)\n",
    "\n",
    "dcid_short = dcid[['initiator_iso', 'target_iso']]\n",
    "# print(dcid_short)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the adjacency matrix for dcid\n",
    "\n",
    "ct = pd.crosstab(dcid_short['initiator_iso'], dcid_short['target_iso'])\n",
    "idx = ct.columns.union(ct.index)\n",
    "adjacency_matrix = ct.reindex(index = idx, columns=idx, fill_value=0)\n",
    "\n",
    "print(adjacency_matrix)\n",
    "\n",
    "# Save the adjacency matrix to a csv file\n",
    "adjacency_matrix.to_csv('adjacency_matrix.csv', index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if the matrix is symmetric\n",
    "import numpy as np\n",
    "if np.array_equal(adjacency_matrix, adjacency_matrix.T):\n",
    "    print('Matrix is symmetric')\n",
    "else:\n",
    "    print('Matrix is not symmetric')\n",
    "\n",
    "# Save the adjacency matrix to a csv file\n",
    "adjacency_matrix.to_csv('adjacency_matrix.csv') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "G = nx.DiGraph(adjacency_matrix)\n",
    "print(G.edges(data=True))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Draw G graph including labels and weights\n",
    "nx.draw(G, with_labels=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Keep only nodes with outdegree > 0\n",
    "# The node out_degree is the number of edges pointing out of the node\n",
    "# The ones initiating the attack\n",
    "outdeg = G.out_degree()\n",
    "\n",
    "G2 = G.copy()\n",
    "for node in G.nodes():\n",
    "    if outdeg[node] == 0:\n",
    "        G2.remove_node(node)\n",
    "\n",
    "print(G2.edges(data=True))\n",
    "nx.draw(G2, with_labels=True)\n",
    "\n",
    "# Now we can see that some connections are only one way: e.g. \n",
    "# 365 Russia -> 732 (South Korea)\n",
    "# FIXME: not every time I create the graph! WHY IS THIS RANDOM?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Show the weights of the edges\n",
    "pos=nx.spring_layout(G) \n",
    "nx.draw_networkx(G,pos)\n",
    "\n",
    "# Todo: Avoid overlapping of the weight labels\n",
    "# Todo: Avoid overlapping of the nodes\n",
    "labels = nx.get_edge_attributes(G,'weight')\n",
    "nx.draw_networkx_edge_labels(G,pos,edge_labels=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyvis.network import Network\n",
    "\n",
    "net = Network(notebook=True, cdn_resources='in_line')\n",
    "net.from_nx(G)\n",
    "net.show(\"output/network.html\")\n",
    "\n",
    "# # Open the HTML file in the browser or\n",
    "# # read from file to buffer\n",
    "# with open(\"network.html\", \"r\") as file:\n",
    "#     network_html = file.read()\n",
    "\n",
    "# HTML(network_html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make the graph directed\n",
    "\n",
    "net = Network(notebook=True, directed =True, cdn_resources='in_line')\n",
    "net.from_nx(G)\n",
    "net.show(\"output/network-directed.html\")\n",
    "\n",
    "# #read from file to buffer\n",
    "# with open(\"network-directed.html\", \"r\") as file:\n",
    "#     network_html = file.read()\n",
    "\n",
    "# HTML(network_html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Network images\n",
    "![](./output/network-weights.png)\n",
    "![](./output/network-directed.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try curved edges\n",
    "G = nx.DiGraph(adjacency_matrix)\n",
    "print(G.edges(data=True))\n",
    "\n",
    "pos = nx.spring_layout(G)\n",
    "nx.draw(G, with_labels=True, connectionstyle=\"arc3,rad=0.1\")\n",
    "\n",
    "labels = nx.get_edge_attributes(G,'weight')\n",
    "\n",
    "# FIXME: place the labels in a better way\n",
    "nx.draw_networkx_edge_labels(G,pos,edge_labels=labels, label_pos=0.1)\n",
    "plt.show()\n",
    "\n",
    "# show the datastructure of the graph\n",
    "print(G.edges(data=True))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Top Cyber Powers\n",
    "\n",
    "![](./output/cyber-power-index.png)\n",
    "\n",
    "National Cyber Power Index, Voo et al. (2022)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculte some properties of the graph\n",
    "\n",
    "# https://networkx.org/documentation/stable/reference/algorithms/generated/networkx.algorithms.centrality.degree_centrality.html\n",
    "# The degree centrality for a node v is the fraction of nodes it is connected to.\n",
    "centrality = nx.degree_centrality(G)\n",
    "centrality = pd.DataFrame(centrality.items(), columns=['Country', 'Centrality'])\n",
    "centrality = centrality.sort_values(by='Centrality', ascending=False)\n",
    "print(centrality)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Out Degree: https://networkx.org/documentation/stable/reference/classes/generated/networkx.DiGraph.out_degree.html\n",
    "# The node out_degree is the number of edges pointing out of the node. \n",
    "out_degree = G.out_degree()\n",
    "sorted_out_degree = sorted(out_degree, key=lambda x: x[1], reverse=True)\n",
    "sorted_out_degree = pd.DataFrame(sorted_out_degree, columns=['Country', 'Out Degree'])\n",
    "print(sorted_out_degree)\n",
    "\n",
    "# Weighted out degree\n",
    "# The weighted node degree is the sum of the edge weights for edges incident to that node.\n",
    "weighted_out_degree = G.out_degree(weight='weight')\n",
    "sorted_weighted_out_degree = sorted(weighted_out_degree, key=lambda x: x[1], reverse=True)\n",
    "sorted_weighted_out_degree = pd.DataFrame(sorted_weighted_out_degree, columns=['Country', 'Weighted Out Degree'])\n",
    "print(sorted_weighted_out_degree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In Degree: https://networkx.org/documentation/stable/reference/classes/generated/networkx.DiGraph.in_degree.html\n",
    "# The node in_degree is the number of edges pointing to the node. \n",
    "# Number of antagonsits attacking a country\n",
    "in_degree = G.in_degree()\n",
    "sorted_in_degree = sorted(in_degree, key=lambda x: x[1], reverse=True)\n",
    "sorted_in_degree = pd.DataFrame(sorted_in_degree, columns=['Country', 'In Degree'])\n",
    "print(sorted_in_degree)\n",
    "\n",
    "# The weighted node degree is the sum of the edge weights for edges incident to that node.\n",
    "weighted_in_degree = G.in_degree(weight='weight')\n",
    "sorted_weighted_in_degree = sorted(weighted_in_degree, key=lambda x: x[1], reverse=True)\n",
    "sorted_weighted_in_degree = pd.DataFrame(sorted_weighted_in_degree, columns=['Country', 'Weighted In Degree'])\n",
    "print(sorted_weighted_in_degree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reciprocity: https://networkx.org/documentation/stable/reference/algorithms/reciprocity.html\n",
    "# The reciprocity of a directed graph is the ratio of the number of edges pointing in both directions\n",
    "# to the total number of edges in the graph.\n",
    "\n",
    "reciprocity = nx.overall_reciprocity(G)\n",
    "print(reciprocity)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cliques: https://networkx.org/documentation/stable/reference/algorithms/generated/networkx.algorithms.clique.find_cliques.html\n",
    "# A clique in an undirected graph is a subset of the nodes such that every two nodes in the subset are adjacent.\n",
    "\n",
    "# cliques = nx.find_cliques(G)\n",
    "# Only works for an undirected graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "Preparing and reshaping the data took longest. I was struggling with messy data (Excel with multiple working sheets) and merging data to get country codes. Additionally, drawing my particular graph, including the labels on the edges, proved to be more difficult than expected. The graph would look great on a spatial map, but geo-coding the data will take more time.\n",
    "\n",
    "\n",
    "Creating the adjacency matrix for my dataset was easy. Running network analysis and calculating statistics such as degree centrality or reciprocity is super easy after you create the graph.\n",
    "\n",
    "Network analysis is an interesting method, and I'll use it in my research one way or another.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}