generated from roberttwomey/generative-text
-
Notifications
You must be signed in to change notification settings - Fork 1
/
index.html
336 lines (291 loc) · 29.5 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="X-UA-Compatible" content="ie=edge">
<title>A.I. Space Rovers</title>
<link href="https://fonts.googleapis.com/css?family=IBM+Plex+Sans:200,400,600&display=swap" rel="stylesheet">
<link href="https://fonts.googleapis.com/css?family=PT+Serif:400,700&display=swap" rel="stylesheet">
<link rel="stylesheet" href="./styles/style.css">
</head>
<body>
<div class="header">
<div class="title">
<span class="main">
The verbosified work of A.I. Space Rovers
</span>
<br>
<span class="subtitle">
Or: Convincing GPT-2 that more is more
</span>
</div>
</div>
<div class="divider"></div>
<div class="content-wrapper">
<div class="content">
<p>
In early 2019, <a href="https://openai.com/blog/better-language-models/">OpenAI released a language model called GPT-2</a> which was widely <a href="https://www.independent.co.uk/life-style/gadgets-and-tech/news/ai-artificial-intelligence-dangerous-text-gpt2-elon-musk-a9192121.html">sensationalized</a> as being "<a href="https://www.theguardian.com/technology/2019/feb/14/elon-musk-backed-ai-writes-convincing-news-fiction">too dangerous to release</a>" due to its capability to generate bulk amounts of believable text. It's precisely that model which I chose to use as the main tool for this project. This art project: accomplishes what a word-count-seeking writer could only dream of; is a statement to the needlessly verbose writing style I often find myself embodying; and showcases that the 'scariness' of GPT-2 depends largely on how the model is configured.
</p>
<div id="description-container" class="collapsed">
<div id="desc-toggle"><a>…</a></div>
<p>
To generate text using GPT-2, the user passes a prompt to the model, and the model attempts to thematically continue the given text. The result is very often creative and produces interesting narratives that develop the prompt into a full-fledged story. Such an example, featured on OpenAI's blog post, is shown below. The first two sentences are the prompt.
</p>
<blockquote>
<div class="prompt">
<p>
In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
</p>
</div>
<div class="output">
<p>
The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.
</p>
<p>
Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.
</p>
<p>
Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.
</p>
</div>
</blockquote>
<p style="text-align: center;">…</p>
<p>
Whereas GPT-2 has a commendable creative ability, I aimed to force the model to artistically overfit on the prompt and produce fluff—new sentences which add almost nothing of value to the overall essay. I essentially turned the once "dangerous" AI into a highschool student desperate to reach a word count. Below, I took one of my own highschool essays and doubled the word count using an instance of GPT-2 which I finetuned one-at-a-time on three sentence excerpts from my essay. For each finetuned model I then produced and handpicked my favorite samples.
</p>
<p>
The non-finetuned GPT-2 model released by OpenAI has a notable habit of introducing new topics if it doesn't have adequate knowledge of the subject included in the prompt. Despite my tinkering to avoid this, there are still a handful of samples in which the model managed to break free from the restrictions I placed on it. These occurrences lead to some... let's say... interesting results.
</p>
<p>
Sections of text below that look
<span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span>
<!-- <span class="generated-text">like this</span> -->
were generated by the machine. These sections are followed by a dropdown where you can choose to look at another one of my favorite generated samples. You can also use the 'preset' buttons to quickly view the essay with filler text, with some interesting off-topic narrative, or without any generated text.
</p>
</div>
</div>
</div>
<div class="divider"></div>
<div class="paper">
<div class="page">
<p class="not-body">
Parker Grey Addison
</p>
<p class="not-body">
Ms. Wolper
</p>
<p class="not-body">
Science Fiction, 4°
</p>
<p class="not-body">
03.28.2016
</p>
<p>
<h3>
Artificial Intelligence in Planetary Rovers
</h3>
</p>
<p>
As of 1970, when the Soviet Union’s Lunokhod, the first unmanned rover, landed on the Moon, rovers have been agreed on as one of the best ways to gather scientific data about celestial bodies outside of our own Earth. Rovers, by nature not needing a human present, are able to travel to and survive on planets and moons not suitable for humans. Not only do they not need the same accommodations as humans, they also: last much longer than a human would on another planet, can transfer huge amounts of scientific data back to scientists on Earth, and they don’t necessarily need to have a plan to return to Earth. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> While starting off as directly remote controlled devices with limited data collection ability, planetary and lunar rovers have advanced greatly, and gained the capability of many more actions. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Now, rovers can collect precise measurements, perform highly specific and intricate tasks, and even carry out entire scientific operations—ranging from basic to complex—solely on their own. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> A huge amount of this advancement can be contributed not to progress in equipment, but instead progress in the Artificial Intelligence the rovers possess. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Just a few years after the initial use of rovers on other celestial bodies, space programs in the Soviet Union, the U.S., and all over the globe, saw the importance of autonomous capabilities on their rovers. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Ever since that discovery, progress on this A.I. has been steadily increasing.
</p>
<p>
On November 17, 1970—a year after the Apollo 11 moon landing—the first unmanned space rover made contact with the lunar surface. This rover was the Lunokhod 1, from the Soviet Union’s Luna program. A couple months later, on January 15, 1973, another Soviet rover, Lunokhod 2, made a successful landing. The rovers were equipped with cameras and some basic scientific instruments meant to measure the composition and properties of the lunar soil.[1] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Another year later, in December of 1971, the Soviet Mars 3 lander was making its descent onto the planet. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> However, as a result of a dust storm on the surface, a connection with the lander was only able to be sustained for 20 seconds. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> The lander contained a vehicle on board, called the Prop-M rover. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> While ultimately unsuccessful, the Prop-M rover was the first rover to have a form of artificial intelligence. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> This A.I. was very limited, and consisted merely of two metal rods on the front of the rover that would allow of autonomous obstacle avoidance.[2] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> For the period from the 1970’s until the turn of the century, A.I. in space rovers was no more than simple obstacle avoidance. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> With the launch of NASA’s two “robotic geologists”[3] , a huge shift occurred in the software of rovers.
</p>
<p>
The Spirit and Opportunity rovers, part of NASA’s Mars Exploration Rovers program, were launched on July 7 and 10, 2003, and landed on Mars on January 3 and 24, 2004. Featuring still limited artificial intelligence, these rovers were expected to only have a three month lifespan to collect and transmit data. Lasting long past their expected terminations—Spirit ceased to function as of 2010, but Opportunity is still functioning without issue—the two rovers have underwent a great deal of improvements.[3] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Rovers are generally thought of as having cutting-edge technology in order to make scientific discoveries and advancements. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Due to the length of development time and travel time, the building phase lasting over 8 years for NASA’s Curiosity, equipment onboard the rovers is generally robust, but outdated. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> While this equipment is something that cannot be easily upgraded once the rover has left Earth, the programming of the rover can be changed, and updated.[4] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> A large part of the updates to Discovery have been towards increasing the rover’s artificial intelligence. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> The same process is also in place for other rovers and machines, such as Curiosity and other planned rovers.[5] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> The benefits of this increased intelligence are numerous. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> On a basic level, the A.I. assists with navigation. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> The rovers can avoid obstacles and can find better paths to and from ordered positions with greater ease and safety, all with less input needed from the humans back on Earth. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> As software is updated, and intelligence is added, the rovers’ capabilities drastically increase.[6] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> For data collection, rovers are smarter about knowing when to transfer data back to Earth. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> While in the past some data has been lost due to poor connection, or not enough storage and being overwritten, the rovers now can decide when they should backup data according to amount of storage left, and broadband connectivity with mission control.[7] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> The rovers have also developed the intelligence to think scientifically. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> If a photo that they take, or object that they see is deemed scientifically interesting, they will seek closer inspection and take more pictures with different angles and filters. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> If the object is unimportant—an empty sky devoid of storms or interesting features, for example—then they will only store it temporarily, and won’t hesitate to overwrite it. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Unless, of course, they are instructed otherwise.[6][8] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> All in all, as rovers become more autonomous, they are able to do their job quicker, instead of waiting for a command that can take upwards of 10 minutes to be received. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> As more rovers are sent to Mars and to other planets, it is entirely possible that there will be interconnected teams of rovers working together as scientists. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Eventually, barely any human moderation will be needed for entire experiments and research projects to take place.[5]
</p>
<p>
Increased artificial intelligence for planetary rovers is not without sceptics. Indeed, there are plenty of obstacles and possible issues that could arise before the A.I. in the rovers can work as desired. A logistical problem that A.I. faces is the numerous unintended and unforeseen consequences of small errors in the programming, and the sheer amount of time it takes to correct these small errors. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> While the rovers, and the new software updates go through many tests before completion, and while there hasn’t yet been a large mishap relating to any updates, the possibility still exists. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Especially as the intelligence gets more complex, it is likely for there to be situations in which the code doesn’t “think” the way we want it to.[9] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> The second main issue with increasing the A.I. of rovers stems from the fear a large proportion of people have about artificial intelligence. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> While A.I. is commonly accepted as benign, advanced but still limited, there is still great uncertainty about the future of its advancements. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> A great deal of people have concerns about creating something that, not only can think, but that can think differently than we do.[4] <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> In order to address this fear, people must be guaranteed that the rovers will ultimately follow orders from humans. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> If the A.I. were to progress to superiority over humans, then there is a possibility the rover may ignore human input, unless programmed to ensure otherwise. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> Even then, there is a lot that a rover could do in the 10 to 20 minutes it takes for communication transmissions to be sent from Mars to Earth and back again. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> However, that doesn’t seem to bother too many scientists and engineers.
</p>
<p>
Since the turn of the century, artificial intelligence has become increasingly present in a wide variety of machines and robotics. Among experimental robots, Planetary rovers—developed for the most part by organizations such as NASA and the European Space Agency—have been at the cutting edge of A.I. advancements. The A.I. assists with many aspects of the rovers’ roles, allowing for more efficient basic functions, as well as scientifically thoughtful analysis and scientifically prioritized decisions that can be made autonomously. <span class="generated"><span class="generated-text"></span><span class="dropdown"><select></select></span></span> While there are dissenters to the increasing intelligence of rovers, primarily due to a fear of a created thought process, the development of A.I. is continuing to advance and is continuing to be uploaded to rovers on Mars.
</p>
</div>
<div class="menu">
<p>
Presets:
</p>
<div class="button">
<button id="0">
Fluffy
</button>
(mundane filler text)
</div>
<div class="button">
<button id="1">
Strange
</button>
(creative narratives...)
</div>
<div class="button">
<button id="2">
Sparse
</button>
(no generated text)
</div>
</div>
</div>
<div class="divider"></div>
<div class="content-wrapper">
<div class="content">
<h2>
For the technically inclined
</h2>
<p>
To create the art piece seen above, I finetuned and explored OpenAI's 124M model for GPT-2. I also leveraged <a href="https://github.com/minimaxir/gpt-2-simple">Max Woolf's GPT-2-simple package</a> for Python to ease the finetuning process.
</p>
<p>
I wanted to purposefully misuse parts of the GPT-2 model in order to put an end to model's creativity, and to produce something in stark contrast to the text-summarative capabilities that GPT-2 has been cited for. This involved a large amount of experimentation with different generation parameters such as temperature and top_k, as well as exploration of the use and over-use of finetuning.
</p>
</div>
</div>
<div class="content-wrapper bg-slate">
<div class="content">
<h3>
I tried different methods of <code><b>finetuning</b></code> with the following findings:
</h3>
<ol>
<li>
Finetuning on a large corpus of academic writing, magazine writing, and newspaper writing (medium amount of epochs, ~500)
<ul>
<li>
The model had a strong desire to talk about topics mentioned in this corpus—did not stay on topic to the prompt
</li>
</ul>
</li>
<li>
Finetuning on the essay that I wanted to verbosify (small amount of epochs, ~1–5)
<ul>
<li>
The model generally stayed on topic, but the specific subject discussed still had decent variability
</li>
</ul>
</li>
<li>
Finetuning on (1) followed by (2)
<ul>
<li>
The model would generally stay on topic, but would often get derailed by topics from (1)
</li>
</ul>
</li>
<li>
Finetuning on the prompt that I wanted to verbosity (smallest amount of epochs, only 1)
<ul>
<li>
AMAZING! The model would generally output very similar information to that contained by the prompt, and would occasionally rely on general knowledge to complete an idea. I ended up solely using this
</li>
</ul>
</li>
</ol>
</div>
</div>
<div class="content-wrapper bg-mars">
<div class="content">
<h3>
I tried different <code><b>parameters</b></code> with the following findings:
</h3>
<ol>
<li>
Temperature - affects variability of output
<ul>
<li>
Low temperatures (0.1–0.4) resulted in low variability (duh) and high likelihood of the output getting caught in a cycle. If the text was off-subject, setting a low temperature did not fix the issue, and instead only cemented the output as slightly off-subject
</li>
<li>
Medium temperatures (0.4–0.7) resulted in moderate variability and low likelihood of the output getting caught in a cycle. This often lead to the output repeating the prompt if the model didn't know much about the subject being discussed
</li>
<li>
High temperatures (0.7–1.0) resulted in good variability. I used primarily a temperature of 0.8, as this allowed for the model to reference its general knowledge about the subject being discussed. If many samples were returning blank or returning a limited output, then bumping the temperature up allowed the model to attempt to bring up a different subject
</li>
</ul>
</li>
<li>
Top-k - allows the model to choose from only the top-k tokens during a generation step
<ul>
<li>
Default (0) was horrible since it actually removes all restrictions and allows sampling from all tokens. This meant that the model went off topic very easily
</li>
<li>
Recommended (40) was generally good, since it restricted the model to generating words that were seen in or similar to those seen in the prompt (since I also finetuned on the prompt)
</li>
<li>
Lower (30) was what I used for most of my generations, since it further restricted the model to using words similar to those in the prompt. Lower top-k forces the model to stay on subject, since it isn't allowed to mention unrelated words. A low top-k can be used with a high temperature to very effectively repeat the finetuning data/prompt while still attempting to avoid replication
</li>
<li>
Higher (80) was what I used if many samples were returning blank or the model was having a hard time avoiding replication of the prompt. Doing this returned a bit of creative freedom to the model and allowed it to rely on general knowledge to continue the prompt
</li>
</ul>
</li>
<li>
Length - determines how many tokens are generated
<ul>
<li>
I generated 100 tokens for each sample, since putting too small a size often cut off sentences that seemed to have good potential, and too large a size goes off topic or enters a cycle
</li>
</ul>
</li>
</ol>
</div>
</div>
<div class="content-wrapper">
<div class="content">
<p>
I played around with these finetuning methods and parameters on a handful of prompts from high school essays, and then got to work doubling the word count of one essay in particular. In order to do this, I created a collection of "passages" that formed the essay. Each passage consisted of three consecutive sentences that were all contained in the same paragraph. (I also played around with using three versus four sentences, but found that three gave better results for my particular high-school-writing-style.) Then, for each passage, I initialized a GPT-2 model and ran one iteration of finetuning on the passage. I then generated a handful of samples, using the passage as the prompt. I would vary the parameters in accordance to how I explain them above. Finally, I would hand pick some of my favorite generated sentences and add those to my essay. I found good sentences for most passages using only 10 samples, though some I had to look at up to 40 samples (playing with parameters every 10) in order to find something worthwhile. Note that the only modifications I made to the samples was adding a period if the output had been truncated.
</p>
<p>
The end result is (in my opinion) beautiful. For the most part, the sentences that I selected are very mundane. They don't pull the essay off topic, they don't detract from its value, and they certainly don't add to it... with the exception being that they add to the word count! However, there are some instances—particularly in the first few passages—where the creative side of GPT-2 was trying as hard as it could to reveal itself... these instances produced some interesting, amusing, and sometimes terrifying outputs.
</p>
<p>
Overall, this was a time consuming process when done on an entire essay, but I could see it being somewhat useful, or at least somewhat amusing as an extension to Google Docs. After all, the only thing the model needs to be finetuned on is the prompt itself, so the training and generation is actually quite quick and very accessible—no need for a large corpus, just a few sentences.
</p>
</div>
</div>
<div class="footer">
<p>
©2019 <a href="https://parkeraddison.com">Parker Grey Addison</a>
|
<a href="https://github.com/parkeraddison/aiArt-verbosified-space-rovers"><code></></code> on GitHub</a>
</p>
<p>
Photo by <a href="https://unsplash.com/@lobosnico?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Nicolas Lobos</a> on Unsplash
</p>
</div>
</body>
<script src="./scripts/main.js"></script>
<script>
let desc = document.getElementById("description-container");
function toggleDesc() {
desc.classList.toggle("collapsed");
}
let descToggle = document.getElementById("desc-toggle");
descToggle.addEventListener("click", toggleDesc);
// "curlies.js" courtesy of Ali Karbassi
// https://gist.github.com/karbassi/6216484
function curlies(element) {
function smarten(text) {
return text
/* opening singles */
.replace(/(^|[-\u2014\s(\["])'/g, "$1\u2018")
/* closing singles & apostrophes */
.replace(/'/g, "\u2019")
/* opening doubles */
.replace(/(^|[-\u2014/\[(\u2018\s])"/g, "$1\u201c")
/* closing doubles */
.replace(/"/g, "\u201d")
/* em-dashes */
.replace(/--/g, "\u2014");
};
var children = element.children;
if (children.length) {
for(var i = 0, l = children.length; i < l; i++) {
curlies(children[i]);
}
} else {
element.innerHTML = smarten(element.innerHTML);
}
};
curlies(document.body);
</script>
</html>