-
Notifications
You must be signed in to change notification settings - Fork 80
/
faq.html
384 lines (350 loc) · 21.2 KB
/
faq.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>FAQ - OpenTSDB - A Distributed, Scalable Monitoring System</title>
<!-- Bootstrap core CSS -->
<link href="css/bootstrap.min.css" rel="stylesheet">
<link href="css/otsdb-bootstrap.css" rel="stylesheet">
<!-- Custom styles for this template -->
<link href="css/custom.css" rel="stylesheet">
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', 'UA-18339382-1']);
_gaq.push(['_setDomainName', 'none']);
_gaq.push(['_setAllowLinker', true]);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
</head>
<body>
<a href="https://github.com/OpenTSDB/opentsdb">
<img style="position: absolute; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/38ef81f8aca64bb9a64448d0d70f1308ef5341ab/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f6461726b626c75655f3132313632312e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png">
</a>
<div class="container">
<h1><a href="http://www.opentsdb.net/"><img src="img/logo_header.png"></a></h1>
<div class="navbar navbar-default">
<div class="container-fluid">
<div class="navbar-header">
<ul class="nav navbar-nav">
<li><a href="index.html">Home</a></li>
<li><a href="overview.html">Overview</a></li>
<li><a href="https://github.com/OpenTSDB/opentsdb/releases">Download</a></li>
<li><a href="https://github.com/OpenTSDB/opentsdb">Source</a></li>
<li><a href="docs/build/html/index.html">Documentation</a></li>
<li><a href="faq.html">FAQ</a></li>
</ul>
</div>
</div>
</div><script src="misc/toc.js" type="text/javascript"></script>
<section id="FAQ">
<div id="toc"></div>
<a name="scalability"></a><h2>Scalability</h2>
<h4>Can OpenTSDB scale to multiple data centers?</h4>
Yes. It is recommended that you run one set of Time Series Daemons
(TSDs) per HBase cluster and one HBase cluster per physical datacenter.
It is not recommended to have HBase clusters spanning across data
centers. Instead you can use
<a href="http://hbase.apache.org/replication.html">HBase replication</a>
to replicate tables across data centers.
<h4>How much <em>write</em> throughput can I get with OpenTSDB?</h4>
It depends mostly on two things:
<ol><li>The size of your HBase cluster.</li><li>The CPUs you're using.</li></ol>
If your HBase cluster is reasonably sized, it's unlikely that OpenTSDB will
max it out as the TSDs tend to be CPU bound before that happens (unless you
run many TSDs). A TSD can easily handle 2000 new data points per second per
core on an old dual-core Intel Xeon CPU from 2006. More modern CPUs will get
you more throughput.
<h4>How much <em>read</em> throughput can I get with OpenTSDB?</h4>
Read throughput varies depending on the cardinality of a metric (how many
distinct time series exist), the time span and the number of data points retreived.
Lower cardinality with fewer data points will execute much quicker than higher
cardinality and greater data point queries. Most queries for the last day of data
will return in less than a second with low cardinality. However huge queries can
run for multiple seconds. We're working to optimize the query path.
<h4>What type of hardware should I run the TSDs on?</h4>
There are no strict requirements. The recommended configuration, however, is
a 4-core machine with at least 4GB of RAM, and a <code>tmpfs</code> partition
for the cache directory used by the TSD. Having more RAM helps the TSD ride
over transient HBase outages by allowing it to buffer more incoming data
before getting to the point where it must start discarding data.
<p>
HBase region servers are usually beefy machines and many OpenTSDB users run
their TSDs on the same machines, granted there is enough memory for both processes.
<h4>How much disk space do I need?</h4>
The answer depends mostly on the average number of tags per data point.
StumbleUpon uses 4.5 tags on average and 100+ billion data points take
only just over a terabyte of disk space (pre-HDFS 3x replication). Enabling compression
with <a href="setup-hbase.html#lzo">LZO</a> or Snappy is
extremely recommended in a production setting. In Stumbleupon's case, each data point
ends up taking about 12 bytes of disk space (or actually 36 if you include the
3x replication factor of HDFS). We also find that, on average, LZO is able to
achieve a compression factor of 4.2x on the TSD table, but your mileage will vary.
Without LZO, a data point costs roughly: 16 bytes of HBase overhead, 3 bytes
for the metric, 4 bytes for the timestamp, 6 bytes per tag, 2 bytes of
OpenTSDB overhead, up to 8 bytes for the value. Integers are stored with variable
length encoding and can consume 1, 2, 4 or 8 bytes.
<a name="reliability"></a><h2>Reliability</h2>
<h4>What are the Single Points of Failure of OpenTSDB?</h4>
OpenTSDB itself doesn't have any specific SPoF as there is no central
component and you can run multiple TSDs on different machines. The TSDs
need HBase to run, and HBase doesn't have any SPoF<sup>*</sup> either as HBase
only really needs a ZooKeeper quorum to keep serving. A ZooKeeper quorum is
typically made of 5 different machines, out of which you can afford to lose 2
before the system goes down. Note that although HBase has a master, it is not
actually needed for HBase to keep serving. Not having a master running will
prevent HBase from starting or recovering from machine failures but, in a
steady state, losing the master doesn't impede on HBase's ability to serve.
<p>
<small><sup>*</sup> Fine prints: if your HBase cluster is backed by HDFS,
which is most likely the case for production clusters at the time of this
writing, then you have a SPoF because of the NameNode of HDFS. If you run
HBase on top of a <em>reliable</em> distributed filesystem, then you don't
have any SPoF.</small>
<h4>What are the failure modes of OpenTSDB?</h4>
The TSD eventually becomes unhealthy when HBase itself is down or broken for
an extended period of time. Right now, the TSD doesn't handle
prolonged HBase outages very well and will discard incoming data points once its buffers
are full if it's unable to flush them to HBase. Future versions will temporarily store
data to local disk when it is unable to reach HBase.
<p>
StumbleUpon has had a number of cases where a collector that runs on
hundreds of machines goes crazy and generate a DDoS on the TSDs. The TSD
doesn't do a good job at handling such DDoS situations by penalizing offending
clients, so its performance will degrade once the machine it's running on is
unable to keep up with the load.
<h4>What is the recommended deployment for OpenTSDB?</h4>
We recommend that you run multiple TSDs behind a load balancer such as Varnish,
HAProxy or DNS round robin. StumbleUpon found it useful
to dedicate one or more TSDs for read queries (human users using the web UI to
generate graphs or to view dashboards) and let other TSDs handle the write
queries (new data points coming in from production machines). For the
"read-only" TSDs, we recommend Varnish for load balancing.
<a href="varnish.html">Read more about Varnish and TSDs</a>.
<h4>What data durability guarantees does OpenTSDB make?</h4>
By default the TSD buffers data points for about 1 second before persisting
them in HBase (configurable via the <code>--flush-interval</code> flag). If
the TSD was to crash without getting a chance to run its shutdown hook, you
could lose up to 1 second worth of data points. In practice we've found this
trade off to be acceptable given the performance benefits that deferred
flushes offer in terms of write throughput. Once a data point has been stored
in HBase, data durability is guaranteed if you're running HBase on top of a
distributed filesystem that provides the necessary data durability guarantees.
<p>
If you use HDFS, we recommend that you run
<a href="http://www.cloudera.com/downloads/">Cloudera's Distribution for
Hadoop</a> (CDH), version 3 or above preferably, as this version comes with
all the necessary patches to make HDFS less unreliable and has better
performance.
<a name="datamodel"></a><h2>Data Model</h2>
<h4>How can I increment a counter in OpenTSDB?</h4>
OpenTSDB does not have a counter feature at this time, though work is underway.
Currently OpenTSDB simply records <code>(timestamp,
value)</code> pairs. Data points are independent from each other. Say you
want to keep track of clicks on an ad in OpenTSDB. You wouldn't send a "+1"
to the TSD for every click. Instead, if your application doesn't already
keeps track of click counts, you'd need to increment a counter for every click
and periodically send the value of that counter to the TSD. You can later
query the TSD and ask for the rate of change of the counter, which will give
you clicks per second.
<h4>Can I store sub-second precision timestamps in OpenTSDB?</h4>
As of version 2.0 you can store data with millisecond timestamps. However
we recommend you avoid storing a data point at each millisecond as this will
slow down queries dramatically. See
<a href="docs/build/html/user_guide/query/dates.html">Dates and Times</a>
for details.
<h4>Can I use another storage backend than HBase?</h4>
Not at this time. OpenTSDB was designed specifically for a storage backend that follows
the <a href="http://labs.google.com/papers/bigtable.html">Bigtable</a> data
model (a distributed, strongly consistent, sorted multi-dimensional hash map).
At the time OpenTSDB was written, HBase is the only such system that's both open-source
and usable in production, so the code was written specifically for HBase.
Technically it would be feasible to port the code to other systems that follow
the Bigtable data model. Systems that differ by not storing data in a sorted
fashion (such as distributed hash tables) or that do not offer a strong
consistency guarantee will simply not work with the current design.
<p>
We're looking at adding support for Cassandra now that it implements counters.
<a name="misc"></a><h2>Misc</h2>
<h4>How do the TSDs handle DST changes or leap seconds?</h4>
The TSD doesn't assign timestamps to your data points, your collectors do.
It is strongly recommended that you use UNIX timestamps in your collectors,
so all your timestamps will be based on
<a href="http://en.wikipedia.org/wiki/Unix_epoch">Epoch</a>. This way
you will not be affected by timezone adjustments or DST changes on your
machines.
<p>
The TSD always renders timestamps in local time when using the GUI, to make it easier for us
human to understand and correlate events based on the timezone we live in.
So you should to make sure you give the TSD the correct timezone setting
(e.g. via the <a href="cli.html#Overriding_the_timezone_of_the_TSD"><code>TZ</code> environment variable</a>).
When the TSD starts,
it computes its offset from UTC and will then keep that offset forever.
In case of a DST change, for instance, it would then appear that the TSD
is 1 hour behind. There are plans to periodically re-compute the offset
from UTC to avoid that situation, but right now you have to restart the
TSD in order to adjust the offset. Note that this doesn't prevent the TSD
from working properly, it only affects anything that parses dates from local
time or renders them in local time. Dashboards and alerting systems should
use relative time (e.g. "1d ago") and should thus be unaffected.
<p>
When <a href="http://en.wikipedia.org/wiki/Leap_second">leap</a> seconds
occur, UNIX timestamps go back by one second. The TSD should handle this
situation gracefully (although this hasn't been tested yet). Unless you're
collecting data every second, you won't notice anything except that the
interval between the two data points where the leap second occurred is one
second less than it should have been. If you do collect data every second,
the second data point that attempts to overwrite the previous one during the
leap second will be discarded with an error message.
<h4>The graphs are ugly, can they be made prettier?</h4>
Ugliness is a subjective thing :)
<p>
There are a lot of knobs that aren't exposed yet that would allow the TSD to
generate nicer, antialiased, smoothed graphs. It's just a matter of exposing
those Gnuplot knobs. Also, recent versions of Gnuplot can generate graphs in
HTML5 canvas. We plan to use this to build pretty graphs you can interact
with from your web browser.
<p>
Please contribute to help make the UI sexier.
<h4>Can I use OpenTSDB to generate graphs for my customers / end-users?</h4>
Yes, but you have to be careful with that. OpenTSDB was written for internal
use only, to help engineers and operations staff understand and manage large
computer systems. It hasn't been through any security review and does not
included authentication.
<p>
We don't recommend that you give direct access to the TSD to untrusted users.
If you really want to leverage the TSD's graphing features, we recommend that
you put the TSD behind a secured HTTP proxy that only allows specific requests
to go through. Alternatively, you could use the TSD to periodically
pre-generate a fixed set of graphs and serve them as static images to your
customers.
<h4>Why does OpenTSDB return more data than I asked for in my query?</h4>
All queries specify a start time and an end time (if the end time isn't
specified, it is assumed to be "now"). OpenTSDB's goal is to plot a sensible
graph covering that time span. However it needs to retrieve data before and
after the times you actually specified, in order to know how to properly
compute the values near the "edges" of the graph. Because having extra values
past the times actually requested is required to draw accurate graphs,
OpenTSDB also returns the extra data based on the assumption that if you want
to plot your own graphs or make your own processing, you will also need the
extra data to get the correct behavior near the edges.
<p>
The amount of extra data that OpenTSDB attempts to retrieve is proportional to
the time span covered by your query. The 2.0 HTTP API will only return data within
the requested time span.
<h4>I don't understand the data points returned for my query</h4>
Sometimes the results to a query don't match people's expectations. This is
often because it's not necesssarily quite obvious what steps are involved in
a query, why OpenTSDB uses interpolation, when do aggregators kick in.
Please see the documentation for
<a href="http://opentsdb.net/docs/build/html/user_guide/query/aggregators.html">Aggregators</a>
for details.
<a name="meta"></a><h2>Meta</h2>
<h4>Why was OpenTSDB written in Java?</h4>
Mostly because OpenTSDB lives around the HBase community, which is a Java
community. OpenTSDB also started by using HBase's library, which only
exists in Java. Eventually, however, OpenTSDB started to use another
alternative library to access HBase
(<a href="http://github.com/OpenTSDB/asynchbase">asynchbase</a>)
but sadly this one too is in Java.
<h4>Why HBase and not Cassandra</h4>
Early on Cassandra didn't support atomic operations required in the assignment of UIDs.
Now that Cassandra has such features, including counters, it would be possible to copy
the HBase schema in Cassandra, though there are some differences to work through and
an asynchronous driver may need to be developed.
<h4>Has OpenTSDB anything to do with OpenBSD?</h4>
While the author of OpenTSDB admires the work done on OpenBSD, the fact that
the name of projects are so close is just a coincidence. "TSDB" alone was
too ambiguous, and the author miserably failed to come up with a better name.
<h4>Who is behind OpenTSDB?</h4>
OpenTSDB was originally designed and implemented at
<a href="http://www.stumbleupon.com">StumbleUpon</a> by Benoît Sigoure.
Berk D. Demir contributed ideas and feedback during the early design stages.
<a href="https://github.com/OpenTSDB/tcollector"><code>tcollector</code></a>
was designed and implemented by Mark Smith and Dave
Barr, with contributions from Benoît Sigoure.
<h4>How to contribute to OpenTSDB?</h4>
The easiest way is to <a href="http://help.github.com/fork-a-repo/">fork</a>
the project on <a href="https://github.com/OpenTSDB/opentsdb">GitHub</a>.
Make whatever changes you want to your own fork, then send a
<a href="http://help.github.com/send-pull-requests/">pull request</a>.
You can also send your patches to the
<a href="http://groups.google.com/group/opentsdb">mailing list</a>.
Be prepared to go through a couple iterations as the code is being reviewed
before getting accepted in the main repository. If you are familiar with
how the Linux kernel is developped, then this is pretty similar.
<h4>Who commits to OpenTSDB?</h4>
Anyone can commit to OpenTSDB, provided that the changes are accepted
in the main repository after getting reviewed. There is no notion of
"commit access", or no list of committers.
<h4>Why does OpenTSDB use the LGPL?</h4>
One of the most frequent "holy war" that plague open-source communities is
that of what licenses to use, which ones are better or "more free" than others.
OpenTSDB uses the <a href="http://www.gnu.org/licenses/lgpl.html">GNU LGPLv2.1+</a>
for maximum compatibility with its dependencies and other licenses, and
because its author thinks that the LGPL strikes the right balance between the
goals of free software and the legal restrictions often present in corporate
environments.
<p>
Let's stress the following points:
<ul>
<li>The LGPL is <em>not</em> the GPL. Although based on the same text, the
way it extends the GPL has significant consequences. Do not confuse the two.</li>
<li>The <a href="http://www.gnu.org/licenses/lgpl-java.html">LGPL is perfectly
compatible with Java</a>. The myth that the LGPL does not work as intended
with Java is, well, just a myth, albeit a widespread one.</li>
<li>The LGPL allows you to use the code in proprietary software, provided that
you don't <em>redistribute a modified version</em> of the LGPL'ed code.</li>
<li>If you want to redistribute a modified version of the code, then your
changes must be released under the LGPL.</li>
<li>The LGPL is perfectly compatible with the
<a href="http://www.apache.org/licenses/LICENSE-2.0">ASF2</a> license.
Many people are misled to believe that there is an incompatibility because the
Apache Software Foundation (ASF) decided to not allow inclusion of LGPL'ed
code in its own projects. This choice only applies to the projects managed by
the ASF itself and doesn't stem from any license incompability.</li>
</ul>
With this out of the way, we hope that those afraid of the 3 letters "GPL"
will acknowledge the importance of using the LGPL in OpenTSDB and will
overcome their fear of the license.
<br/>
<small>Disclaimer: This page doesn't provide any formal legal advice.
Information given here is given in good faith. If you have any doubt, talk to
a lawyer first. In the text above "LGPL" or "GPL" refers to the version 2.1 of
the license, or (at your option) any later version.</small>
<h4>Who supports OpenTSDB?</h4>
<a href="http://www.stumbleupon.jobs">StumbleUpon</a> supported the initial
development of OpenTSDB as well as its open-source release.
<a href="http://www.yahoo.com">Yahoo!</a> maintained OpenTSDB until 2021. It is
currently in maintenance mode as the number of time series storage solutions have
exploded.
Many other engineers and companies have contributed to OpenTSDB over the years.
Please see our <a href="https://github.com/OpenTSDB/opentsdb/blob/next/THANKS">
thanks</a> file for contributors. And thank you to everyone who has made OpenTSDB
so popular and successful.
<p>
YourKit is kindly supporting open source projects with its full-featured Java
Profiler. YourKit, LLC is the creator of innovative and intelligent tools for
profiling Java and .NET applications. Take a look at YourKit's leading
software products:
<a href="http://www.yourkit.com/java/profiler/index.jsp">YourKit Java Profiler</a>
and
<a href="http://www.yourkit.com/.net/profiler/index.jsp">YourKit .NET Profiler</a>.
</section>
<hr>
<div class="footer">
<p><a href="https://groups.google.com/forum/#!forum/opentsdb"><i class="glyphicon glyphicon-envelope"></i> Mailing List</a> | <i class="glyphicon glyphicon-user"></i> IRC: Freenode #opentsdb | © 2010 - <script type="text/JavaScript"> document.write(new Date().getFullYear()); </script> The OpenTSDB Authors | Travis CI Status: <img src="https://travis-ci.org/OpenTSDB/opentsdb.svg?branch=master">
</div>
</div> <!-- /container -->
<!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
<!-- Include all compiled plugins (below), or include individual files as needed -->
<script src="js/bootstrap.min.js"></script>
</body>
</html>