chore(docs): developer docs on plugin system communication

site/content/docs/contributing/_index.md

@@ -30,4 +30,12 @@ A walkthrough of running Hipcheck for the first time.
 A walkthrough of running Hipcheck for the first time.
 {% end %}
 
+{% waypoint(title="PR Review Checklist", path="@/docs/contributing/pr-review.md", icon="message-circle") %}
+Things to track when reviewing a PR.
+{% end %}
+
+{% waypoint(title="Developer Docs", path="@/docs/contributing/developer-docs/_index.md") %}
+Documentation for Hipcheck developers.
+{% end %}
+
 </div>

site/content/docs/contributing/developer-docs/_index.md

@@ -0,0 +1,25 @@
+---
+title: Developer Docs
+template: docs.html
+sort_by: weight
+page_template: docs_page.html
+weight: 5
+---
+
+# Hipcheck Developer Docs
+
+<div class="grid grid-cols-2 gap-8 mt-8">
+
+{% waypoint(title="Repo Structure", path="@/docs/contributing/developer-docs/repo-structure.md") %}
+List of key directories in the Hipcheck repository.
+{% end %}
+
+{% waypoint(title="Architecture", path="@/docs/contributing/developer-docs/architecture.md") %}
+Hipcheck's distributed architecture and how plugins get started.
+{% end %}
+
+{% waypoint(title="Query System", path="@/docs/contributing/developer-docs/plugin-query-system/index.md") %}
+The life of a plugin query from inception, through gRPC, to SDK, and back.
+{% end %}
+
+</div>

site/content/docs/contributing/developer-docs/architecture.md

@@ -0,0 +1,57 @@
+---
+title: Architecture
+weight: 2
+---
+
+# Architecture and Plugin Startup
+
+This document describes the distributed architecture of Hipcheck and how plugins
+get started.
+
+Hipcheck is a relatively simple multiprocessed tool that follows a star topology
+(there is a single central node to which all other nodes connect). Users invoke
+the main Hipcheck binary, often referred to as "Hipcheck core" or `hc`, on the
+command line, and provide a [policy file][policy_file] which specifies the set
+of top-level plugins to use during analysis. Once Hipcheck resolves these
+plugins and their dependencies, it starts each plugin in a separate child
+process. Once all plugins are started and initialized, Hipcheck
+core enters the analysis phase. During this phase it acts as a simple hub for
+querying top-level plugins and relaying queries between plugins, as plugins are
+intended to only communicate with each other through the core.
+
+## Plugin Startup
+
+Hipcheck core uses the `plugins/manager.rs::PluginExecutor` struct to start
+plugins. The `PluginExecutor` has fields like `max_spawn_attempts` and
+`backoff_interval` for controlling the startup process. These fields can be
+configured using the `Exec.kdl` file.
+
+The main function in `PluginExecutor` is `start_plugin()`, which takes a
+description of a plugin on file and returns a `Result` containing a handle to
+the plugin process, called `PluginContext`.
+
+In `start_plugin()`, once the `PluginExecutor` has done the work of locating the
+plugin entrypoint binary on disk, it moves into a loop of attempting to start
+the plugin, at most `max_spawn_attempts` times. For each spawn attempt, it will
+call `PluginExecutor::get_available_port()` to get valid local port to tell the
+plugin to listen on. The executor creates a
+`std::process::Command` object for the child process, with `stdout/stderr`
+forwarded to Hipcheck core.
+
+Within each spawn attempt, `PluginExecutor` will try to connect to the port on
+which the plugin should be listening. Since process startup and port
+initialization can take differing amounts of time, the executor does a series of
+up to `max_conn_attempts` connection attempts. For each failed connection,
+the executor waits `backoff_interval`, which increases linearly with the number of
+failed connections. The calculated backoff is modulated by a random `jitter`
+between 0 and `jitter_percent`.
+
+Overall, the sleep duration between failed connections is equal to
+
+	(backoff * conn_attempts) * (1.0 +/- jitter)
+
+As soon as `PluginExecutor::start_plugin()` successfully starts and connects to the child
+process, it stores the process and plugin information in a `PluginContext` and
+returns it to the caller. It however returns an error if `max_spawn_attempts` is reached.
+
+[policy_file]: @/docs/guide/config/policy-file.md

site/content/docs/contributing/developer-docs/plugin-query-system/developer_docs_plugin_grpc.drawio

@@ -0,0 +1 @@
+<mxfile host="charts.mitre.org" modified="2024-12-16T18:24:52.210Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Edg/131.0.0.0" version="20.8.5" etag="K-q-bwHREtJwMxqPlC3R" type="device"><diagram id="kli-Tog3HTVBQzar_-YL" name="Page-1">7Vxrc+K2Gv41zLQfyPgCBj4mJDl7prd0t+e0/ZQRtsDqGotIdoD99dXVNxnigsHebHdmJ7Ysy+LV+7x3aeDO17v/ELAJf8IBjAaOFewG7v3AcRxrPGN/eMtetkwd1bAiKJBNdt7wCX2BqtFSrSkKIC11TDCOErQpN/o4jqGflNoAIXhb7rbEUfmrG7CCRsMnH0Rm6+8oSEL1K8ZW3v4BolWov2xb6ska6M6qgYYgwNtCk/swcOcE40RerXdzGHHiabrI9x4PPM0mRmCcNHnh6Zffv4DlyPdmT788v/rYI3QxHI/kMK8gStUvVrNN9poEBKdxAPko1sC924YogZ82wOdPt2zRWVuYrCN2Z7PLJYqiOY4wEe+6wRhOgxFrpwnBn2HhydRZuJ7HnqgJQJLA3cGfZmcEY5wG8RomZM+6aDbzFI0Vk9kjdb/Nl2yi1yEsLNdYvwgUm6yysXNKsgtFzH9AWHt6YcICOF36dYT1/ClcLNsh7DDj3iOUHU2vSlnPIOz/KOS/iuBXxMnJLtNFhGgICf9+LFqidIViPnS8xEKGDPgv8yI23bsF6+etEkFk0dvHhHBhwn4Sez2C4j3+VsgvKeRP8JIP7ifoFdYOJb9IsxksMdkCwq8GnvWSQkYJTiL1lF1/hqqFzc29NZgFBkwuqVtMkhCvcAyih7z1rsxOeZ8fMd4oJvoLJsleCVmQ8g8VWUx+k3/oOLuweeGU+PAY+ytRDcgKJsfkTz37ERgBQdmSOG8dowYnwZitGbwhlK+Ae/vBfxAN8k4s2nffnwfjFkA5KkPSmZqQdKwaSI4uhUhTh/w3gQQkHBj4VaAQAj8UrL0ZRvBV2AiAseaeIlqLngAyKAosF4C3wRHy9xlmmJKOOkIKWymy/4O/fzPWt3+q4cTN/a50t1d3LSLMaYiwA7x0HYQ5BmdQJltzgIm7ZwJpGiX0XGhVNORs9vg4m10EcrbXNeRmBmF/S0ksfxNgn4CECuwIPQfY/1+FxmF4XPwlNFsVcFUlJV5EzKYmAmIeWHNqxwu6ySj6zUFu3BByky4hNzY4Q9s+j2scZMi7FZbLk3zUZ/U2srrGmnZFiySFhEFlzZVQmMafUbzi0IprcCaBxGzTIPXrDUWOzv9DX5uUcknKwzAszzO1R6H4IzVqdTAcQ/1NqTOZHQvRq5xgZgSz6fiQUq2ev1kdOmmqQ89VourVJ8wkcs7pdoXVvSoLy4mptypcnE3jdMaeHBYVyX4DaSYsTD6TXPobATHdsLXvtQQZO51LENPTOJXSN8lOmU5MDPSP1l5NfODKtG4Q0aIh2PDLZQR3tzxIKIRMoC7v/QhQimRwBZBEN8dcuJboVyY23KGkIM/Y3Z9a1LHrXJrxGy3MijYr83lAKlb/kGi03xCNYrqPiBOsbVFpN/XonWk9vxQYwq5jiKzxTKFajQWOZxVOkz/VEKrGQCO3PNDEbiadGbuAfaHbhnegzbXAZGQdnZc7PdqfXcgZtKoqbNOsXH18mgsDCDAvIaq1IH4EC/akBBkQoVXMMcY4kJsed1z6IObK36oHaxQE0sCAFH0BCzEeZ2ZFRzb4+G4wvj8mvlQyQL08yIKYRcY/IjwOh0OtG2tmTUrkd1vhWrs86FDf6xHwcinCju0bAbbpSjLV85LytMTdB7TxQ+h/Zs/nzFPPH5h6Ksy6PlLiG+zAlii5HCdw65JKQ9M2YuJKbhdFrWpqQf8Nq6GBGnfFtq+pAB0zQI5kOI4Kr4AprRWPxQ085RDccLsNQR5/0fFnx5KR8HyRC2upeWCBSQDJ0Jc0vRWfJN8Nh8X27yWv5FwjeGUNEj+UPsnPYA0Dw2liztXFPq4/Jj0jEWNcEsx9udAXsoPzufK1SCoi+CipFW8XdpDqbYMWdbqWXW/r9E7D9HqahvX8XAnXS1O5GLKvEVMiofNMmfuLcPy8BFGEmGTpgSXtVPR6D+L6JuERD1G8KASh9SaCa/bzGBOI4AeBLykiQs4EIAE1+TUJNBDjRGbntBs0F6SiVLybxynyPB7vYI4XM+khn+k55S0SwTyr1mFK7dQgx+Wx3zSY6XpHuda68Txvcp4NdHkjZzT6d/FPWvxZl4J/ZAaotiGMC2hn1qKfRlr8+JgLJGHnzMUkk5Rw/V2VGh9FvmnIpfhG6H/5gWNi4qvzZEbHPRkG25kzKvsc7XgybmlQ172WH+OYHirT7JltIK0+ccmMuh6q++5zio5ZslRvZx0KUj6obrsbZlu/9pDG3UcnXdOkysRUMQsk8zjcC7LW0iEp5YWMtI9OEzH0goJHI3NADSI1HWnCtwKal9eEs6aa0OlSEzpmkOYUaCr5x8D5TMC2hwDtPlXjmqmaHKA8VUtRvIoMQOahCwXAg75PXyB5IPNQm8PI0ha1OYxTwd0mjKcNYeyca9DWGz0TU9HcWIV/TnnApimI6rBj7+iwBxIS13S0XNMKC+BQFEnkpbB0HzMU8HhtffGf0IOq4ijKZZjWffLBSmoCXeNUftjDOqXOVd3ZGqxZOsp1rEZc2RrHOS2rxh6qxe5roEa2QZRvOYDiek3Nxk5rAUemR8fL1qSf8JJCmhgCtFmBKH1nwZI3cnw8yGlVoiXqm+fmfUuDXi3p65o5wtOFZl/KvjLnoT8JlBrddGrl6HXLRoup0F5Ui15epDctBnUPFDhdR6S7h4o2T0FuT2oIq8DtQRSgnVCo7JPsbnpK6O7joXrgf+3KQr7tGnblSaWK2bZ37ZdUt5xX+nvO0f6XKVUcmyH23OilGxxT2MTqbbXm6F0bz1Y9K+bG89Stljcqu+jcUt/SoFfLNGqQvvN6pInXN3N6bHotOdQ+psKb/XT/w7EqVRrwAlUi+lbo+14rVMeTcWkd6ypU605wuNgyeg2OxrjMDo22d1sU1zAfr7oL45R9IW3aFE1zI7bl1jPStbZuTCsSx5tWWLBp3qRqdxgDtbR1o/odna85NK9hRqn6Fy5jEHmmvnqPeze8A375e967oTOARraatS14kb3IsJkBosP1I6Lynk2dYh+BRFTHZeW+QW64dp9nu7gvpsRhI8HZ7TE21qEC+RO2l5LdszrzqE8JsaFj9S1yYeuEQPFwkyyxXQVcHpEtHMphGKe6tMQIBavdKBravStn7zy5rSDYCKyd1jRnE313u+4NjHYfxrWtmr1wXBkOPAXPjzoiJHa+oThg5kxyoCxFlZeLWvN5hsVC7bg8korvVumLOuxBaZZtNT9l40Ak6Uz7zLC6Mya8UomKrTfPX5cljvqnFZe56rNeQWA3ZotuxbWZb/sqDgk0pHH34TvbMgM/zNvjP7PO5smOLqsX1lXprGS3TKhvQ6g2GKqTA/WOwhDQzJDizo0xijDR9EYiYXnRrRhI+EWoUDojvlR9X0xaO0zmNse+6IVWgdw0Z6Ww1B2UD8Xtv5qj0QxQd18YaNecjvbGGY8W/6KBnMtL1cr5kPx0yLbOhzQW5pK7uXgCIzs1XNoo+dnr7sPf</diagram></mxfile>

site/content/docs/contributing/developer-docs/plugin-query-system/developer_docs_plugin_sender_receiver.drawio

@@ -0,0 +1 @@
+<mxfile host="charts.mitre.org" modified="2025-01-13T15:40:41.676Z" agent="5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Edg/131.0.0.0" etag="Fj3XGhVbHTjsAIQxuAur" version="20.8.5" type="device"><diagram id="m5juCMCQGp0zw13HWF7r" name="Page-1">7Vxbc+o2EP41zLQPp+O7zWMglzPTdpoepu3po7EFeGJsKkwC+fWVseSLJLAgli/hvGTs9ZVvv939tJIz0qfr/RN0N6vfYx+EI03x9yP9fqRpY0dHf1PDITNYtpEZljDwM5NaGGbBO8BGBVt3gQ+2lROTOA6TYFM1enEUAS+p2FwI47fqaYs4rD514y4BY5h5bsha/wn8ZJVZHVMp7F9BsFyRJ6sKPrJ2ycnYsF25fvxWMukPI30K4zjJttb7KQhT7Agu2XWPJ47mLwZBlIhc8Kxt/vhrsfDiJ+fL+P3lz9f135MvFvbGqxvu8C/Gb5scCAQw3kU+SO+ijPTJ2ypIwGzjeunRN+RzZFsl6xDtqWiTfSv8oq8AJmBfMuG3fALxGiTwgE4hRw0zuwRTRnMwgm+FA1QD21Yl8C1sc7HPl/mtC1jQBkbmEpR0BqWv3gxst0EczWLvBSQjzQrRW0zmEG0t060ZQKDBj4G5CMJwGocxPF6r+y5wFh6ybxMYv4DSEctzwHzREPwmBf+YA7/Ggd+QBb9p95+kqtU1SYnb+oSSTvLkGZRyRMooqWNZMBla/2Cy7CthkkYmlQHpdhKeaned8AQoitTEJt30djA8TKB79EktvoUz0r3QnYPwOd4GCXIrsoVgkd4jRS5ACug36vA68P308RM3DJapAWZQ5BfcYXt+oh9ApMmyq7fxLgWzEYdZVLjogg7TZDmMow+CjbcC3guy/rcD8PDTz6P06XTQBNE2cSMPKVzawQiLpOo8AruHkEOBdgZ3CLbBuzs/3iqNuk0cRMnxN5uTkXmf3muXIMcexbbKxFcUR4AKRmxqxHnVaBubrO/UNoNNZaOtNtd9Ax4IXgeZ7XQqePKRVmfZTmXl3XO4WwbRQ4T+AHTkrpVqs1gAy+Pi79vjuaI0g7/hUMmr82qjOnX4T9qp9iZwfIOHv6PNdcuSg38P+D+uw3/aDv8dD/D5P3dMw5TEf83sGn/ysIvyj5QK0E4GsnqXgTSL8QCD6y0LXrpm8wSvxms2SBO8GluzGdF0I4LW0CjnKILhpEtzTm1Bv71hh62Z9V4yW/USW/Z/JL1zZarzpKezQqE0zJ/GELSi04CKlLLNUwljy9bdhnSyo/RNJejsXNXnhZ8u+bpoVZEGv8Gy/0e+OuMxTryYvB6+tHRlsE38b7stEmbK7P7Xmyn8FpXHDNYtTptl3xCYpkNJ6y5du5CCH7pIVHunAiZFG+yD5Ds+km7/m9p/MfHe/b502v0B75wEFgUE9MCZ9yfdusSFS5Cc0zcYQeBXVliwjioHCMcTxAZB6CZoyF15XZ578BOeUxIWPMinsUk9o2cYs1+OryqczNxIpybrmADOkGFudGRL/rM/QCCBwXLXE5g5Jp1NYBq1TWVOU1NOW7+VtiatljXROTF5eoEdhX5euUar5R7ALzC87DxNcFZ2tZsmTFbU1vfe5fR+W+m+M2mi+8VdnKUmt5MmuoefXbrA4IpGc6/58OxCbRqhl/xe3imp03S3kKfHPaJPW9a0uFTValqiKnqiaU0qn+bL7S7VtBbVw867DTWaFjHBPZROwwM/CarXFFio/Pl5OhbkKeFBT3hqqSfU0cU8pQZxjDDonqes6v1YnWqg6jAL6jmjs5bXKrPi9HbWl3Y/NrAEelASkqkmI5syrnt8VJTzSyXq86xgmiVLB/uaZmkCdZdmT/TkbH5kFNzO7thohrYE+mcDURIyuG+KaoyekV9zKC7REvbq/i4tVhojP/85cskv8CmVBPLbQ2E/+dJv8OynM/bV7KdryKDZf0VLWkqzr51uE/2djNH5GgJLoCndRPpRByo8DdHiS5jck/Rj05Ps1zaibPMEZRtOP6pCvTHJeydfTOe/mNR8RXJky+EymGgRbdv2LFrGBsU9egWBaLTQaxoMwaUIl0cL3R+2zkeLQ1cetY1oYWeS2oiWwWhbksEGFy5U6r1a2zLhImlkpyp0gNcUF8fmx7HccOlm5k9YiykXhIvvblf5e14fIeS/4QwtQlQqQpjpO9EIoQKEmU9sKEDMs9VEEt37PYHYAdvNgconegmLfSXZ1RNZt2m208Fpt8F2yTM8HbK2X0swmMm9a8UJ06yRJk6qbHRaYCOJ2BIbl9+epwwj2W8QRD8UEP9Wgdeoq5L7JDfFO290O1blrPPirXOU9tWBw1a/z+0B64QEl+ABtFv8F8csZIp/hak//A8=</diagram></mxfile>