Skip to content

Commit

Permalink
rebuilt site
Browse files Browse the repository at this point in the history
  • Loading branch information
PhillHenry committed Dec 6, 2023
1 parent f7f6f68 commit 7e8e1d8
Show file tree
Hide file tree
Showing 4 changed files with 115 additions and 36 deletions.
46 changes: 23 additions & 23 deletions docs/ChangeDataFlow.html
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,9 @@
+-------+-----------------------+------+--------+------------+----------------------------------------------------------------------------------------------------------------+----+--------+---------+-----------+--------------+-------------+------------------------------------------------------------+------------+-----------------------------------+
|version|timestamp |userId|userName|operation |operationParameters |job |notebook|clusterId|readVersion|isolationLevel|isBlindAppend|operationMetrics |userMetadata|engineInfo |
+-------+-----------------------+------+--------+------------+----------------------------------------------------------------------------------------------------------------+----+--------+---------+-----------+--------------+-------------+------------------------------------------------------------+------------+-----------------------------------+
|2 |2023-12-04 16:06:52.87 |NULL |NULL |WRITE |{mode -> Append, partitionBy -> []} |NULL|NULL |NULL |1 |Serializable |true |{numFiles -> 2, numOutputRows -> 20, numOutputBytes -> 3372}|NULL |Apache-Spark/3.5.0 Delta-Lake/3.0.0|
|1 |2023-12-04 16:06:51.39 |NULL |NULL |WRITE |{mode -> Append, partitionBy -> []} |NULL|NULL |NULL |0 |Serializable |true |{numFiles -> 2, numOutputRows -> 20, numOutputBytes -> 3372}|NULL |Apache-Spark/3.5.0 Delta-Lake/3.0.0|
|0 |2023-12-04 16:06:43.094|NULL |NULL |CREATE TABLE|{isManaged -> true, description -> NULL, partitionBy -> [], properties -> {"delta.enableChangeDataFeed":"true"}}|NULL|NULL |NULL |NULL |Serializable |true |{} |NULL |Apache-Spark/3.5.0 Delta-Lake/3.0.0|
|2 |2023-12-06 14:05:09.157|NULL |NULL |WRITE |{mode -> Append, partitionBy -> []} |NULL|NULL |NULL |1 |Serializable |true |{numFiles -> 2, numOutputRows -> 20, numOutputBytes -> 3371}|NULL |Apache-Spark/3.5.0 Delta-Lake/3.0.0|
|1 |2023-12-06 14:05:07.713|NULL |NULL |WRITE |{mode -> Append, partitionBy -> []} |NULL|NULL |NULL |0 |Serializable |true |{numFiles -> 2, numOutputRows -> 20, numOutputBytes -> 3371}|NULL |Apache-Spark/3.5.0 Delta-Lake/3.0.0|
|0 |2023-12-06 14:05:01.405|NULL |NULL |CREATE TABLE|{isManaged -> true, description -> NULL, partitionBy -> [], properties -> {"delta.enableChangeDataFeed":"true"}}|NULL|NULL |NULL |NULL |Serializable |true |{} |NULL |Apache-Spark/3.5.0 Delta-Lake/3.0.0|
+-------+-----------------------+------+--------+------------+----------------------------------------------------------------------------------------------------------------+----+--------+---------+-----------+--------------+-------------+------------------------------------------------------------+------------+-----------------------------------+


Expand All @@ -88,26 +88,26 @@
+---+-------+------------+----------+-----------------------+
|id |label |partitionKey|date |timestamp |
+---+-------+------------+----------+-----------------------+
|0 |label_0|0 |2023-12-04|2023-12-04 16:06:34.216|
|0 |label_0|0 |2023-12-04|2023-12-04 16:06:34.216|
|1 |label_1|1 |2023-12-03|2023-12-04 16:06:34.416|
|1 |label_1|1 |2023-12-03|2023-12-04 16:06:34.416|
|2 |label_2|2 |2023-12-02|2023-12-04 16:06:34.616|
|2 |label_2|2 |2023-12-02|2023-12-04 16:06:34.616|
|3 |label_3|3 |2023-12-01|2023-12-04 16:06:34.816|
|3 |label_3|3 |2023-12-01|2023-12-04 16:06:34.816|
|4 |label_4|4 |2023-11-30|2023-12-04 16:06:35.016|
|4 |label_4|4 |2023-11-30|2023-12-04 16:06:35.016|
|5 |label_5|0 |2023-11-29|2023-12-04 16:06:35.216|
|5 |label_5|0 |2023-11-29|2023-12-04 16:06:35.216|
|6 |label_6|1 |2023-11-28|2023-12-04 16:06:35.416|
|6 |label_6|1 |2023-11-28|2023-12-04 16:06:35.416|
|7 |label_7|2 |2023-11-27|2023-12-04 16:06:35.616|
|7 |label_7|2 |2023-11-27|2023-12-04 16:06:35.616|
|8 |label_8|3 |2023-11-26|2023-12-04 16:06:35.816|
|8 |label_8|3 |2023-11-26|2023-12-04 16:06:35.816|
|9 |label_9|4 |2023-11-25|2023-12-04 16:06:36.016|
|9 |label_9|4 |2023-11-25|2023-12-04 16:06:36.016|
|0 |label_0|0 |2023-12-06|2023-12-06 14:04:52.936|
|0 |label_0|0 |2023-12-06|2023-12-06 14:04:52.936|
|1 |label_1|1 |2023-12-05|2023-12-06 14:04:53.136|
|1 |label_1|1 |2023-12-05|2023-12-06 14:04:53.136|
|2 |label_2|2 |2023-12-04|2023-12-06 14:04:53.336|
|2 |label_2|2 |2023-12-04|2023-12-06 14:04:53.336|
|3 |label_3|3 |2023-12-03|2023-12-06 14:04:53.536|
|3 |label_3|3 |2023-12-03|2023-12-06 14:04:53.536|
|4 |label_4|4 |2023-12-02|2023-12-06 14:04:53.736|
|4 |label_4|4 |2023-12-02|2023-12-06 14:04:53.736|
|5 |label_5|0 |2023-12-01|2023-12-06 14:04:53.936|
|5 |label_5|0 |2023-12-01|2023-12-06 14:04:53.936|
|6 |label_6|1 |2023-11-30|2023-12-06 14:04:54.136|
|6 |label_6|1 |2023-11-30|2023-12-06 14:04:54.136|
|7 |label_7|2 |2023-11-29|2023-12-06 14:04:54.336|
|7 |label_7|2 |2023-11-29|2023-12-06 14:04:54.336|
|8 |label_8|3 |2023-11-28|2023-12-06 14:04:54.536|
|8 |label_8|3 |2023-11-28|2023-12-06 14:04:54.536|
|9 |label_9|4 |2023-11-27|2023-12-06 14:04:54.736|
|9 |label_9|4 |2023-11-27|2023-12-06 14:04:54.736|
+---+-------+------------+----------+-----------------------+
only showing top 20 rows

Expand Down
11 changes: 3 additions & 8 deletions docs/ChangeDataFlowStreaming.html
Original file line number Diff line number Diff line change
Expand Up @@ -68,15 +68,10 @@
) USING DELTA</b>
+ When we start streaming from ChangeDataFlowStreamingSpec to streamsink with a watermark of 4 seconds and a trigger processing time of 4000 ms
+ And the initial count in streamsink is 0
+ And we append 100 rows with a timestamp ranging from 2023-12-04 16:07:02.778 to 2023-12-04 16:08:41.778
+ And we append 100 rows with a timestamp ranging from 2023-12-06 14:05:20.663 to 2023-12-06 14:06:59.663
+ And we wait 4000 ms
+ Then the final row count at Mon Dec 04 16:07:14 UTC 2023 in streamsink is 100 rows
+ Then the final row count at Wed Dec 06 14:05:32 UTC 2023 in streamsink is 100 rows
+ <b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b>
+ <b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b>
Run completed in 41 seconds, 205 milliseconds.
Total number of tests run: 3
Suites: completed 3, aborted 0
Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
All tests passed.</pre>
+ <b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b> </pre>
</body>
</html>
71 changes: 71 additions & 0 deletions docs/Crud.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
<!DOCTYPE html>
<html>
<head>
<style type="text/css">
body {background-color: black;}
pre {
font-weight: normal;
color: #bbb;
white-space: -moz-pre-wrap;
white-space: -o-pre-wrap;
white-space: -pre-wrap;
white-space: pre-wrap;
word-wrap: break-word;
overflow-wrap: break-word;
}
b {font-weight: normal}
b.BOLD {color: #fff}
b.ITA {font-style: italic}
b.UND {text-decoration: underline}
b.STR {text-decoration: line-through}
b.UNDSTR {text-decoration: underline line-through}
b.BLK {color: #000000}
b.RED {color: #aa0000}
b.GRN {color: #00aa00}
b.YEL {color: #aa5500}
b.BLU {color: #0000aa}
b.MAG {color: #aa00aa}
b.CYN {color: #00aaaa}
b.WHI {color: #aaaaaa}
b.HIK {color: #555555}
b.HIR {color: #ff5555}
b.HIG {color: #55ff55}
b.HIY {color: #ffff55}
b.HIB {color: #5555ff}
b.HIM {color: #ff55ff}
b.HIC {color: #55ffff}
b.HIW {color: #ffffff}
b.BBLK {background-color: #000000}
b.BRED {background-color: #aa0000}
b.BGRN {background-color: #00aa00}
b.BYEL {background-color: #aa5500}
b.BBLU {background-color: #0000aa}
b.BMAG {background-color: #aa00aa}
b.BCYN {background-color: #00aaaa}
b.BWHI {background-color: #aaaaaa}
</style>
</head>
<body>
<pre>CrudSpec:
A Delta table
- should be created and populated
+ Given a table created with SQL
<b class=YEL>CREATE TABLE CrudSpec (
id int,
label String,
partitionKey long,
date Date,
timestamp Timestamp
) USING DELTA</b>
+ When we write 20 rows to CrudSpec
+ Then the table indeed contains all the data
+ And the metastore contains a reference to the table CrudSpec
+ <b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b>
+ <b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b><b class=WHI>+ </b><b class=CYN>+ </b><b class=BLU>+ </b><b class=RED>+ </b><b class=GRN>+ </b><b class=MAG>+ </b><b class=YEL>+ </b>
Run completed in 42 seconds, 645 milliseconds.
Total number of tests run: 4
Suites: completed 3, aborted 0
Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0
All tests passed.</pre>
</body>
</html>
23 changes: 18 additions & 5 deletions docs/index.html
Original file line number Diff line number Diff line change
@@ -1,15 +1,17 @@
<!DOCTYPE html>
<html><head>
<meta name="generator" content="Hugo 0.120.4">
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no" />

<title>DeltaLake BDDs</title>
<title> - DeltaLake BDDs</title>



<meta name="description" content="" />
<meta name="description" content="DeltaLake Playground These are BDD (Behaviour Driven Design) tests that both test the code and generate human readable documentation. The code for these tests can be found in GitHub
ChangeDataFlow
ChangeDataFlowStreaming
Crud" />
<meta name="author" content="" />


Expand Down Expand Up @@ -53,7 +55,11 @@
</nav>
<div class="d:flex flex:column@<=sm pt:90 px:24 jc:center gap:44 word-break:break-word">
<div class="max-w:700 w:full box:content-box">
<div class="d:flex flex:col"><div class="
<article class="box:border-box pt:32">
<header class="mb:32">
<div class="font:40 font:extrabold"></div>
<div class="mt:16 f:fade-60"></div>
</header><div class="
_:where(a):hover{text-decoration-color:fade}
_:where(a){text-decoration:2;underline;fade-10;_text-decoration-color:fade-70@dark}
_:where(blockquote){bl:5;solid;fade-76/.1;_bl:5;solid;fade-34/.1@dark}
Expand Down Expand Up @@ -90,7 +96,14 @@
_:where(video,img){max-width:full}
_:where(a,mark){text-underline-offset:3}
_:where(hr){h:2;_bg:fade-10;_bg:fade-70@dark;_my:3em}
"></div><h2 class="my:1em">Posts</h2></div>
"><h2 id="deltalake-playground">DeltaLake Playground</h2>
<p>These are BDD (Behaviour Driven Design) tests that both test
the code and generate human readable documentation.
The code for these tests can be found in <a href="https://github.com/PhillHenry/DeltaLakePlayground">GitHub</a></p>
<p><a href="ChangeDataFlow.html">ChangeDataFlow</a></p>
<p><a href="ChangeDataFlowStreaming.html">ChangeDataFlowStreaming</a></p>
<p><a href="Crud.html">Crud</a></p>
</div></article>
<footer class="py:24">
<div class="f:fade-30 f:14 mb:8"></div>
<div class="f:fade-60 f:12">Theme <a class="f:bold" href="https://github.com/serkodev/holy" _target="_blank">Holy</a></div>
Expand Down

0 comments on commit 7e8e1d8

Please sign in to comment.