Datadog 101 - Building Great Dashboards

# Building Great Dashboards
---
# Two Types Of Dashboards
![](../../images/buildinggreatdashboards/2dashboards.png)

---
# Two Sets of Widgets

### Timeboards
![](../../images/buildinggreatdashboards/timeboardwidgets.png)
### Screenboards
![](../../images/buildinggreatdashboards/screenboardwidgets.png)

---
# Two Different Goals
![](../../images/buildinggreatdashboards/dashboards.png)

---
# ElasticSearch Metrics
![](../../images/buildinggreatdashboards/elasticsearch-alerts-final2.png)

---

# ElasticSearch Metrics

| Metric | Description|
|------|--------| 
| elasticsearch.active_primary_shards (gauge) | The number of active primary shards in the cluster. shown as shard |
| elasticsearch.active_shards (gauge) | The number of active shards in the cluster. shown as shard |
| elasticsearch.breakers.fielddata.estimated_size_in_bytes (gauge) | The estimated size in bytes of the field data circuit breaker. shown as byte |
| elasticsearch.breakers.fielddata.overhead (gauge) | The constant multiplier for byte estimations of the field data circuit breaker. |
| elasticsearch.breakers.fielddata.tripped (gauge) | The number of times the field data circuit breaker has tripped. |
| elasticsearch.breakers.parent.estimated_size_in_bytes (gauge) | The estimated size in bytes of the parent circuit breaker. shown as byte |
| elasticsearch.breakers.parent.overhead (gauge) | The constant multiplier for byte estimations of the parent circuit breaker. |
| elasticsearch.breakers.parent.tripped (gauge) | The number of times the parent circuit breaker has tripped. |
| elasticsearch.breakers.request.estimated_size_in_bytes (gauge) | The estimated size in bytes of the request circuit breaker. shown as byte |
| elasticsearch.breakers.request.overhead (gauge) | The constant multiplier for byte estimations of the request circuit breaker. |
| elasticsearch.breakers.request.tripped (gauge) | The number of times the request circuit breaker has tripped. |

---

| Metric | Description|
|------|--------| 
| elasticsearch.cache.field.evictions (gauge) | The total number of evictions from the field data cache. shown as eviction |
| elasticsearch.cache.field.size (gauge) | The size of the field cache. shown as byte |
| elasticsearch.cache.filter.count (gauge) | The number of items in the filter cache. shown as item |
| elasticsearch.cache.filter.evictions (gauge) | The total number of evictions from the filter cache. shown as eviction |
| elasticsearch.cache.filter.size (gauge) | The size of the filter cache. shown as byte |
| elasticsearch.cluster_status (gauge) | The elasticsearch cluster health as a number: red = 0, yellow = 1, green = 2 |
| elasticsearch.docs.count (gauge) | The total number of documents in the cluster across all shards. shown as document |
| elasticsearch.docs.deleted (gauge) | The total number of documents deleted from the cluster across all shards. shown as document |
| elasticsearch.fielddata.evictions (gauge) | The total number of evictions from the fielddata cache. shown as eviction |
| elasticsearch.fielddata.size (gauge) | The size of the fielddata cache. shown as byte |
| elasticsearch.flush.total (gauge) | The total number of index flushes to disk since start. shown as flush |
| elasticsearch.flush.total.time (gauge) | The total time spent flushing the index to disk. shown as second |

---

| Metric | Description|
|------|--------| 
| elasticsearch.fs.total.available_in_bytes (gauge) | The total number of bytes available to this Java virtual machine on this file store. shown as byte |
| elasticsearch.fs.total.disk_io_op (gauge) | The total I/O operations on the file store. shown as operation |
| elasticsearch.fs.total.disk_io_size_in_bytes (gauge) | Total bytes used for all I/O operations on the file store. shown as byte |
| elasticsearch.fs.total.disk_read_size_in_bytes (gauge) | The total bytes read from the file store. shown as byte |
| elasticsearch.fs.total.disk_reads (gauge) | The total number of reads from the file store. shown as read |
| elasticsearch.fs.total.disk_write_size_in_bytes (gauge) | The total bytes written to the file store. shown as byte |
| elasticsearch.fs.total.disk_writes (gauge) | The total number of writes to the file store. shown as write |
| elasticsearch.fs.total.free_in_bytes (gauge) | The total number of unallocated bytes in the file store. shown as byte |
| elasticsearch.fs.total.total_in_bytes (gauge) | The total size in bytes of the file store. shown as byte |
| elasticsearch.get.current (gauge) | The number of get requests currently running. shown as request |
| elasticsearch.get.exists.time (gauge) | The total time spent on get requests where the document existed. shown as second |
| elasticsearch.get.exists.total (gauge) | The total number of get requests where the document existed. shown as request |
| elasticsearch.get.missing.time (gauge) | The total time spent on get requests where the document was missing. shown as second |
| elasticsearch.get.missing.total (gauge) | The total number of get requests where the document was missing. shown as request |
| elasticsearch.get.time (gauge) | The total time spent on get requests. shown as second |
| elasticsearch.get.total (gauge) | The total number of get requests. shown as request |

---

| Metric | Description|
|------|--------| 
| elasticsearch.http.current_open (gauge) | The number of current open HTTP connections. shown as connection |
| elasticsearch.http.total_opened (gauge) | The total number of opened HTTP connections. shown as connection |
| elasticsearch.id_cache.size (gauge) | The size of the id cache shown as byte |
| elasticsearch.indexing.delete.current (gauge) | The number of documents currently being deleted from an index. shown as document |
| elasticsearch.indexing.delete.time (gauge) | The total time spent deleting documents from an index. shown as second |
| elasticsearch.indexing.delete.total (gauge) | The total number of documents deleted from an index. shown as document |
| elasticsearch.indexing.index.current (gauge) | The number of documents currently being indexed to an index. shown as document |
| elasticsearch.indexing.index.time (gauge) | The total time spent indexing documents to an index. shown as second |
| elasticsearch.indexing.index.total (gauge) | The total number of documents indexed to an index. shown as document |

See all of [the metrics for Elasticsearch](/summit-training-session/notes/buildinggreatdashboards/)
---
# What do you think the work metrics should be?
The server does something.
* **Throughput Metrics**: How many of those things did it do?
* **Performance Metrics**: How long does it take to do those things?
* **Success Metrics**: How many of those things were done successfully?
* **Error Metrics**: How many of those things were not done successfully?

[Review the notes for this section to see them all](../../notes/buildinggreatdashboards/)
---
# What do you think the resource metrics should be?
The server does something and relies on a resource.

* **Utilization Metrics**: How busy was the resource while the server did its thing?
* **Saturation Metrics**: How long is the queue of things waiting for the server?
* **Error Metrics**: How many errors is the resource experiencing with regards to the server?
* **Availability Metrics**: How long is the resource available for?

[Review the notes for this section to see them all](../../notes/buildinggreatdashboards/)
---
# Some Key Elasticsearch Metrics
| Category | Metric |
|------|-----|
|Work:**Throughput**|`elasticsearch.primaries.search.query.total`|
|Work:**Throughput**|`elasticsearch.primaries.search.query.current`|
|Work:**Throughput**|`elasticsearch.primaries.search.fetch.current`|
|Work:**Performance**|`elasticsearch.primaries.search.query.time`|
|Resource:**Utilization**| `jvm.mem.non_heap_committed`|
|Resource:**Utilization**| `jvm.mem.heap_committed`|
|Resource:**Availability**| `elasticsearch.initializing_shards`|
|Resource:**Saturation**| `elasticsearch.thread_pool.bulk.queue`|

See https://www.datadoghq.com/blog/monitor-elasticsearch-performance-metrics/ for more

---
# Building a Timeseries on a Dashboard
1.  Create a Timeboard or Screenboard
1.  Drag Timeseries to the canvas
  1.  Select the metric
  1.  Set the Scope (host, etc)
  1.  Space Aggregation (avg, max, min, sum; by x)
  1.  Time Aggregation (optional)(rollup)
  1.  Functions
---
# Building a Timeseries

![](https://d1ax1i5f2y3x71.cloudfront.net/items/2S173y0a2f0K0B1D2V0d/%5Bc16f26b2d4d9f41c6acabffc2da22d67%5D_Image%25202017-09-25%2520at%252010.44.56%2520AM.public.png)

---
# Building a Timeseries
![](https://cl.ly/0A1Y1q0m453b/Image%202017-09-25%20at%2010.50.57%20AM.public.png)
---
# Building a Timeseries

![](https://cl.ly/3r3d1C1o1T2a/Image%202017-09-25%20at%2010.52.49%20AM.public.png)
---
# Building a Timeseries

![](https://cl.ly/0j3h2g1j2F23/Image%202017-09-25%20at%2010.54.23%20AM.public.png)
---
# Building a Timeseries
![](https://cl.ly/3q1d1V0h0R0u/Image%202017-09-25%20at%2010.56.17%20AM.public.png)

Back to the Agenda

Notes | Hands On Instruction

Session Agenda

Presentations

Notes

Hands On