diff --git a/README.md b/README.md index 59572212ad..269fc57562 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@
-[Distributed TSDB](https://m3db.io/docs/m3db/) and [Query Engine](https://m3db.io/docs/how_to/query/), [Prometheus Sidecar](https://m3db.io/docs/integrations/prometheus/), [Metrics Aggregator](https://m3db.io/docs/overview/components/#m3-aggregator), and more such as [Graphite storage and query engine](https://m3db.io/docs/integrations/graphite/). +[Distributed TSDB](https://m3db.io/docs/reference/m3db/) and [Query Engine](https://m3db.io/docs/how_to/query/), [Prometheus Sidecar](https://m3db.io/docs/integrations/prometheus/), [Metrics Aggregator](https://m3db.io/docs/overview/reference/#m3-aggregator), and more such as [Graphite storage and query engine](https://m3db.io/docs/integrations/graphite/). ## Table of Contents diff --git a/m3query_intro.md b/m3query_intro.md new file mode 100644 index 0000000000..d67072766b --- /dev/null +++ b/m3query_intro.md @@ -0,0 +1,3 @@ +M3 Query is a distributed query engine for querying realtime and historical data stored in M3DB nodes, supporting several query languages. It is designed to support both low latency realtime queries and queries that can take longer to execute, aggregating over larger datasets for analytical use cases. + +For example, if you are using the Prometheus remote write endpoint with [M3 Coordinator](/docs/integrations/prometheus), you can use M3 Query instead of the Prometheus remote read endpoint. By doing so, you get all the benefits of M3 Query's engine such as [block processing](/docs/architecture/m3query/blocks/). As M3 Query provides a Prometheus compatible API, you can use 3rd party graphing and alerting solutions like Grafana. diff --git a/netlify.toml b/netlify.toml index 25f08eec41..95a4fda848 100644 --- a/netlify.toml +++ b/netlify.toml @@ -40,6 +40,14 @@ from="/docs/quickstart/kubernetes" to="/docs/cluster/kubernetes_cluster" +[[redirects]] + from="/docs/how_to/aggregator" + to="/docs/m3aggregator/setup" + +[[redirects]] + from="/docs/how_to/query" + to="/docs/m3query/setup" + # TODO: Fix this with new content type [[redirects]] from = "/talks" diff --git a/site/content/_index.md b/site/content/_index.md index cfebb18ed8..e49731a905 100644 --- a/site/content/_index.md +++ b/site/content/_index.md @@ -15,9 +15,9 @@ other supporting infrastructure. M3 has several features, provided as discrete components, which make it an ideal platform for time series data at scale: -- A distributed time series database, [M3DB](/docs/m3db/), that provides scalable storage for time series data and a reverse index. +- A distributed time series database, [M3DB](/docs/reference/m3db/), that provides scalable storage for time series data and a reverse index. - A sidecar process, [M3Coordinator](/docs/integrations/prometheus), that allows M3DB to act as the long-term storage for Prometheus. -- A distributed query engine, [M3Query](/docs/m3query), with native support for PromQL and Graphite (M3QL coming soon). +- A distributed query engine, [M3Query](/docs/reference/m3query), with native support for PromQL and Graphite (M3QL coming soon). - An aggregation tier, M3Aggregator, that runs as a dedicated metrics aggregator/downsampler allowing metrics to be stored at various retentions at different resolutions. @@ -30,7 +30,7 @@ Getting started with M3 is as easy as following one of the How-To guides. - [Single M3DB node deployment](/docs/quickstart) - [Clustered M3DB deployment](/docs/cluster) - [M3DB on Kubernetes](/docs/operator) -- [Isolated M3Query on deployment](/docs/how_to/query) +- [Isolated M3Query on deployment](/docs/how_to/m3query) ## Support diff --git a/site/content/architecture/_index.md b/site/content/architecture/_index.md new file mode 100644 index 0000000000..8474ba3769 --- /dev/null +++ b/site/content/architecture/_index.md @@ -0,0 +1,4 @@ +--- +title: Architecture +weight: 20 +--- diff --git a/site/content/architecture/m3coordinator.md b/site/content/architecture/m3coordinator.md new file mode 100644 index 0000000000..dce7f746a2 --- /dev/null +++ b/site/content/architecture/m3coordinator.md @@ -0,0 +1,12 @@ +--- +title: "M3 Coordinator" +weight: 3 +--- + +M3 Coordinator is a service that coordinates reads and writes between upstream systems, such as Prometheus, and downstream systems, such as M3DB. + +It also provides management APIs to setup and configure different parts of M3. + +The coordinator is generally a bridge for read and writing different types of metrics formats and a management layer for M3. + +**Note**: M3DB by default includes the M3 Coordinator accessible on port 7201. For production deployments it is recommended to deploy it as a dedicated service to ensure you can scale the write coordination role separately and independently to database nodes as an isolated application separate from the M3DB database role. diff --git a/site/content/architecture/m3db/_index.md b/site/content/architecture/m3db/_index.md new file mode 100644 index 0000000000..4f51344cb9 --- /dev/null +++ b/site/content/architecture/m3db/_index.md @@ -0,0 +1,5 @@ +--- +title: "M3DB, a distributed time series database" +menuTitle: "M3DB" +weight: 1 +--- \ No newline at end of file diff --git a/site/content/m3db/architecture/caching.md b/site/content/architecture/m3db/caching.md similarity index 92% rename from site/content/m3db/architecture/caching.md rename to site/content/architecture/m3db/caching.md index a2352e8843..1d3451fbe5 100644 --- a/site/content/m3db/architecture/caching.md +++ b/site/content/architecture/m3db/caching.md @@ -5,7 +5,7 @@ weight: 7 ## Overview -Blocks that are still being actively compressed / M3TSZ encoded must be kept in memory until they are sealed and flushed to disk. Blocks that have already been sealed, however, don't need to remain in-memory. In order to support efficient reads, M3DB implements various caching policies which determine which flushed blocks are kept in memory, and which are not. The "cache" itself is not a separate datastructure in memory, cached blocks are simply stored in their respective [in-memory objects](/docs/m3db/architecture/engine#in-memory-object-layout) with various different mechanisms (depending on the chosen cache policy) determining which series / blocks are evicted and which are retained. +Blocks that are still being actively compressed / M3TSZ encoded must be kept in memory until they are sealed and flushed to disk. Blocks that have already been sealed, however, don't need to remain in-memory. In order to support efficient reads, M3DB implements various caching policies which determine which flushed blocks are kept in memory, and which are not. The "cache" itself is not a separate datastructure in memory, cached blocks are simply stored in their respective [in-memory objects](/docs/architecture/m3db/engine#in-memory-object-layout) with various different mechanisms (depending on the chosen cache policy) determining which series / blocks are evicted and which are retained. For general purpose workloads, the `lru` caching policy is reccommended. @@ -21,7 +21,7 @@ The `all` cache policy is the opposite of the `none` cache policy. All blocks ar The `recently_read` cache policy keeps all blocks that are read from disk in memory for a configurable duration of time. For example, if the `recently_read` cache policy is set with a duration of 10 minutes, then everytime a block is read from disk it will be kept in memory for at least 10 minutes. This policy can be very effective if only a small portion of your overall dataset is ever read, and especially if that subset is read frequently (i.e as is common in the case of database backing an automatic alerting system), but it can cause very high memory usage during workloads that involve sequentially scanning all of the data. -Data eviction from memory is triggered by the "ticking" process described in the [background processes section](/docs/m3db/architecture/engine#background-processes) +Data eviction from memory is triggered by the "ticking" process described in the [background processes section](/docs/architecture/m3db/engine#background-processes) ## Least Recently Used (LRU) Cache Policy diff --git a/site/content/m3db/architecture/commitlogs.md b/site/content/architecture/m3db/commitlogs.md similarity index 97% rename from site/content/m3db/architecture/commitlogs.md rename to site/content/architecture/m3db/commitlogs.md index c100ed9315..400f2647f5 100644 --- a/site/content/m3db/architecture/commitlogs.md +++ b/site/content/architecture/m3db/commitlogs.md @@ -53,7 +53,7 @@ CommitLogMetadata { ### Compaction / Snapshotting -Commit log files are compacted via the snapshotting proccess which (if enabled at the namespace level) will snapshot all data in memory into compressed files which have the same structure as the [fileset files](/docs/m3db/architecture/storage) but are stored in a different location. Once these snapshot files are created, then all the commit log files whose data are captured by the snapshot files can be deleted. This can result in significant disk savings for M3DB nodes running with large block sizes and high write volume where the size of the (uncompressed) commit logs can quickly get out of hand. +Commit log files are compacted via the snapshotting proccess which (if enabled at the namespace level) will snapshot all data in memory into compressed files which have the same structure as the [fileset files](/docs/architecture/m3db/storage) but are stored in a different location. Once these snapshot files are created, then all the commit log files whose data are captured by the snapshot files can be deleted. This can result in significant disk savings for M3DB nodes running with large block sizes and high write volume where the size of the (uncompressed) commit logs can quickly get out of hand. In addition, since the snapshot files are already compressed, bootstrapping from them is much faster than bootstrapping from raw commit log files because the individual datapoints don't need to be decoded and then M3TSZ encoded. The M3DB node just needs to read the raw bytes off disk and load them into memory. diff --git a/site/content/m3db/architecture/consistencylevels.md b/site/content/architecture/m3db/consistencylevels.md similarity index 100% rename from site/content/m3db/architecture/consistencylevels.md rename to site/content/architecture/m3db/consistencylevels.md diff --git a/site/content/m3db/architecture/engine.md b/site/content/architecture/m3db/engine.md similarity index 96% rename from site/content/m3db/architecture/engine.md rename to site/content/architecture/m3db/engine.md index 2ebf97a07c..224f994c7d 100644 --- a/site/content/m3db/architecture/engine.md +++ b/site/content/architecture/m3db/engine.md @@ -66,9 +66,9 @@ The in-memory portion of M3DB is implemented via a hierarchy of objects: 1. A `database` of which there is only one per M3DB process. The `database` owns multiple `namespace`s. 2. A `namespace` is similar to a table in other databases. Each `namespace` has a unique name and a set of configuration options, such as data retention and block size (which we will discuss in more detail later). A namespace owns multiple `shard`s. -3. A [`shard`](/docs/m3db/architecture/sharding) is effectively the same as a "virtual shard" in Cassandra in that it provides an arbitrary distribution of time series data via a simple hash of the series ID. A shard owns multiple `series`. +3. A [`shard`](/docs/architecture/m3db/sharding) is effectively the same as a "virtual shard" in Cassandra in that it provides an arbitrary distribution of time series data via a simple hash of the series ID. A shard owns multiple `series`. 4. A `series` represents a sequence of time series datapoints. For example, the CPU utilization for a host could be represented as a series with the ID "host1.system.cpu.utilization" (or "__name__=system_cpu_utilization,host=host1"), and a vector of (TIMESTAMP, CPU_LEVEL) tuples. Visualizing this example in a graph, there would a single line with time on the x-axis and CPU utilization on the y-axis. A `series` owns a `buffer` and any cached `block`s. -5. The `buffer` is where all data that has yet to be written to disk gets stored in memory, for a specific series. This includes both new writes to M3DB and data obtained through [bootstrapping](/docs/operational_guide/bootstrapping_crash_recovery). More details on the [buffer](/docs/m3db/architecture/engine#buffer) is explained below. Upon [flushing](/docs/m3db/architecture/engine#flushing), the buffer creates a `block` of its data to be persisted to disk. +5. The `buffer` is where all data that has yet to be written to disk gets stored in memory, for a specific series. This includes both new writes to M3DB and data obtained through [bootstrapping](/docs/operational_guide/bootstrapping_crash_recovery). More details on the [buffer](/docs/architecture/m3db/engine#buffer) is explained below. Upon [flushing](/docs/architecture/m3db/engine#flushing), the buffer creates a `block` of its data to be persisted to disk. 6. A `block` represents a stream of compressed single time series data for a pre-configured block size, for example, a block could hold data for 6-8PM (block size of two hours). A `block` can arrive directly into the series only as a result of getting cached after a read request. Since blocks are in a compressed format, individual datapoints cannot be read from it. In other words, in order to read a single datapoint, the entire block up to that datapoint needs to be decompressed beforehand. ### Persistent storage @@ -77,16 +77,16 @@ While in-memory databases can be useful (and M3DB supports operating in a memory In addition, with large volumes of data, it becomes prohibitively expensive to keep all the data in memory. This is especially true for monitoring workloads which often follow a "write-once, read-never" pattern where less than a few percent of all the data that's stored is ever read. With that type of workload, it's wasteful to keep all of that data in memory when it could be persisted on disk and retrieved when required. -M3DB takes a two-pronged approach to persistent storage that involves combining a [commit log](/docs/m3db/architecture/commitlogs) for disaster recovery with periodic flushing (writing [fileset files](/docs/m3db/architecture/storage) to disk) for efficient retrieval: +M3DB takes a two-pronged approach to persistent storage that involves combining a [commit log](/docs/architecture/m3db/commitlogs) for disaster recovery with periodic flushing (writing [fileset files](/docs/architecture/m3db/storage) to disk) for efficient retrieval: 1. All writes are persisted to a commit log (the commit log can be configured to fsync every write, or optionally batch writes together which is much faster but leaves open the possibility of small amounts of data loss in the case of a catastrophic failure). The commit log is completely uncompressed and exists only to recover unflushed data in the case of a database shutdown (intentional or not) and is never used to satisfy a read request. -2. Periodically (based on the configured block size), all data in the buffer is flushed to disk as immutable [fileset files](/docs/m3db/architecture/storage). These files are highly compressed and can be indexed into via their complementary index files. Check out the [flushing section](#flushing) to learn more about the background flushing process. +2. Periodically (based on the configured block size), all data in the buffer is flushed to disk as immutable [fileset files](/docs/architecture/m3db/storage). These files are highly compressed and can be indexed into via their complementary index files. Check out the [flushing section](#flushing) to learn more about the background flushing process. The block size parameter is the most important variable that needs to be tuned for a particular workload. A small block size will mean more frequent flushing and a smaller memory footprint for the data that is being actively compressed, but it will also reduce the compression ratio and data will take up more space on disk. If the database is stopped for any reason in between flushes, then when the node is started back up those writes will be recovered by reading the commit log or streaming in the data from a peer responsible for the same shard (if the replication factor is larger than one). -While the fileset files are designed to support efficient data retrieval via the series ID, there is still a heavy cost associated with any query that has to retrieve data from disk because going to disk is always much slower than accessing main memory. To compensate for that, M3DB supports various [caching policies](/docs/m3db/architecture/caching) which can significantly improve the performance of reads by caching data in memory. +While the fileset files are designed to support efficient data retrieval via the series ID, there is still a heavy cost associated with any query that has to retrieve data from disk because going to disk is always much slower than accessing main memory. To compensate for that, M3DB supports various [caching policies](/docs/architecture/m3db/caching) which can significantly improve the performance of reads by caching data in memory. ## Write Path @@ -99,10 +99,10 @@ We now have enough context of M3DB's architecture to discuss the lifecycle of a M3DB will consult the database object to check if the namespace exists, and if it does, then it will hash the series ID to determine which shard it belongs to. If the node receiving the write owns that shard, then it will lookup the series in the shard object. If the series exists, then an encoder in the buffer will encode the datapoint into the compressed stream. If the encoder doesn't exist (no writes for this series have occurred yet as part of this block) then a new encoder will be allocated and it will begin a compressed M3TSZ stream with that datapoint. There is also some additional logic for handling multiple encoders and filesets which is discussed in the [buffer](#buffer) section. -At the same time, the write will be appended to the commit log, which is periodically compacted via a snapshot process. Details of this is outlined in the [commit log](/docs/m3db/architecture/commitlogs) page. +At the same time, the write will be appended to the commit log, which is periodically compacted via a snapshot process. Details of this is outlined in the [commit log](/docs/architecture/m3db/commitlogs) page. {{% notice warning %}} -Regardless of the success or failure of the write in a single node, the client will return a success or failure to the caller for the write based on the configured [consistency level](/docs/m3db/architecture/consistencylevels). +Regardless of the success or failure of the write in a single node, the client will return a success or failure to the caller for the write based on the configured [consistency level](/docs/architecture/m3db/consistencylevels). {{% /notice %}} {{% notice warning %}} @@ -139,7 +139,7 @@ Then M3DB will need to consolidate: 1. The not-yet-sealed block from the buffer (located inside an internal lookup in the Series object) **[6PM - 8PM]** 2. The in-memory cached block (also located inside an internal lookup in the Series object). Since there are also cold writes in this block, the cold writes will be consolidated in memory with data found in the cached block before returning. **[4PM - 6PM]** -3. The block from disk (the block will be retrieved from disk and will then be cached according to the current [caching policy](/docs/m3db/architecture/caching)) **[2PM - 4PM]** +3. The block from disk (the block will be retrieved from disk and will then be cached according to the current [caching policy](/docs/architecture/m3db/caching)) **[2PM - 4PM]** Retrieving blocks from the buffer and in-memory cache is simple, the data is already present in memory and easily accessible via hashmaps keyed by series ID. Retrieving a block from disk is more complicated. The flow for retrieving a block from disk is as follows: @@ -148,7 +148,7 @@ Retrieving blocks from the buffer and in-memory cache is simple, the data is alr 3. Jump to the offset in the index file that we obtained from the binary search in the previous step, and begin scanning forward until we identify the index entry for the series ID we're looking for _or_ we get far enough in the index file that it becomes clear that the ID we're looking for doesn't exist (this is possible because the index file is sorted by ID) 4. Jump to the offset in the data file that we obtained from scanning the index file in the previous step, and begin streaming data. -Once M3DB has retrieved the three blocks from their respective locations in memory / on-disk, it will transmit all of the data back to the client. Whether or not the client returns a success to the caller for the read is dependent on the configured [consistency level](/docs/m3db/architecture/consistencylevels). +Once M3DB has retrieved the three blocks from their respective locations in memory / on-disk, it will transmit all of the data back to the client. Whether or not the client returns a success to the caller for the read is dependent on the configured [consistency level](/docs/architecture/m3db/consistencylevels). **Note:** Since M3DB nodes return compressed blocks (the M3DB client decompresses them), it's not possible to return "partial results" for a given block. If any portion of a read request spans a given block, then that block in its entirety must be transmitted back to the client. In practice, this ends up being not much of an issue because of the high compression ratio that M3DB is able to achieve. @@ -217,7 +217,7 @@ M3DB has a variety of processes that run in the background during normal operati ### Flushing -As discussed in the [architecture](#architecture) section, writes are actively buffered / compressed in memory and the commit log is continuously being written to, but eventually data needs to be flushed to disk in the form of [fileset files](/docs/m3db/architecture/storage) to facilitate efficient storage and retrieval. +As discussed in the [architecture](#architecture) section, writes are actively buffered / compressed in memory and the commit log is continuously being written to, but eventually data needs to be flushed to disk in the form of [fileset files](/docs/architecture/m3db/storage) to facilitate efficient storage and retrieval. This is where the configurable "block size" comes into play. The block size is simply a duration of time dictating how long new writes will be compressed (in a streaming manner) in memory before being flushed to disk. Let's use a block size of two hours as an example. @@ -243,7 +243,7 @@ If there are multiple encoders for a block, they need to be merged before flushi #### Removing expired / flushed series and blocks from memory -Depending on the configured [caching policy](/docs/m3db/architecture/caching), the [in-memory object layout](#in-memory-object-layout) can end up with references to series or data blocks that are expired (have fallen out of the retention period) or no longer needed to be in memory (due to the data being flushed to disk or no longer needing to be cached). The background tick will identify these structures and release them from memory. +Depending on the configured [caching policy](/docs/architecture/m3db/caching), the [in-memory object layout](#in-memory-object-layout) can end up with references to series or data blocks that are expired (have fallen out of the retention period) or no longer needed to be in memory (due to the data being flushed to disk or no longer needing to be cached). The background tick will identify these structures and release them from memory. #### Clean up of expired data diff --git a/site/content/architecture/m3db/overview.md b/site/content/architecture/m3db/overview.md new file mode 100644 index 0000000000..d146958c41 --- /dev/null +++ b/site/content/architecture/m3db/overview.md @@ -0,0 +1,43 @@ +--- +title: "Overview" +weight: 1 +--- + +M3DB is written entirely in Go and does not have any required dependencies. For larger deployments, one may use an etcd cluster to manage M3DB cluster membership and topology definition. + +## High Level Goals + +Some of the high level goals for the project are defined as: + +- **Monitoring support:** M3DB was primarily developed for collecting a high volume of monitoring time series data, distributing the storage in a horizontally scalable manner and most efficiently leveraging the hardware. As such time series that are not read frequently are not kept in memory. +- **Highly configurable:** Provide a high level of configuration to support a wide set of use cases and runtime environments. +- **Variable durability:** Providing variable durability guarantees for the write and read side of storing time series data enables a wider variety of applications to use M3DB. This is why replication is primarily synchronous and is provided with configurable consistency levels, to enable consistent writes and reads. It must be possible to use M3DB with strong guarantees that data was replicated to a quorum of nodes and that the data was durable if desired. + +## About + +M3DB, inspired by [Gorilla][gorilla] and [Cassandra][cassandra], is a distributed time series database released as open source by [Uber Technologies][ubeross]. It can be used for storing realtime metrics at long retention. + +Here are some attributes of the project: + +- Distributed time series storage, single nodes use a WAL commit log and persists time windows per shard independently +- Cluster management built on top of [etcd][etcd] +- Built-in synchronous replication with configurable durability and read consistency (one, majority, all, etc) +- M3TSZ float64 compression inspired by Gorilla TSZ compression, configurable as lossless or lossy +- Arbitrary time precision configurable from seconds to nanoseconds precision, able to switch precision with any write +- Configurable out of order writes, currently limited to the size of the configured time window's block size + +## Current Limitations + +Due to the nature of the requirements for the project, which are primarily to reduce the cost of ingesting and storing billions of timeseries and providing fast scalable reads, there are a few limitations currently that make M3DB not suitable for use as a general purpose time series database. + +The project has aimed to avoid compactions when at all possible, currently the only compactions M3DB performs are in-memory for the mutable compressed time series window (default configured at 2 hours). As such out of order writes are limited to the size of a single compressed time series window. Consequently backfilling large amounts of data is not currently possible. + +The project has also optimized the storage and retrieval of float64 values, as such there is no way to use it as a general time series database of arbitrary data structures just yet. + +[gorilla]: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf + +[cassandra]: http://cassandra.apache.org/ + +[etcd]: https://github.com/etcd-io/etcd + +[ubeross]: http://uber.github.io diff --git a/site/content/m3db/architecture/peer_streaming.md b/site/content/architecture/m3db/peer_streaming.md similarity index 100% rename from site/content/m3db/architecture/peer_streaming.md rename to site/content/architecture/m3db/peer_streaming.md diff --git a/site/content/m3db/architecture/sharding.md b/site/content/architecture/m3db/sharding.md similarity index 100% rename from site/content/m3db/architecture/sharding.md rename to site/content/architecture/m3db/sharding.md diff --git a/site/content/m3db/architecture/storage.md b/site/content/architecture/m3db/storage.md similarity index 100% rename from site/content/m3db/architecture/storage.md rename to site/content/architecture/m3db/storage.md diff --git a/site/content/architecture/m3query/_index.md b/site/content/architecture/m3query/_index.md new file mode 100644 index 0000000000..2e5ea6d72f --- /dev/null +++ b/site/content/architecture/m3query/_index.md @@ -0,0 +1,5 @@ +--- +title: "M3 Query, a stateless query server for M3DB and Prometheus" +menuTitle: "M3 Query" +weight: 2 +--- \ No newline at end of file diff --git a/site/content/m3query/architecture/blocks.md b/site/content/architecture/m3query/blocks.md similarity index 99% rename from site/content/m3query/architecture/blocks.md rename to site/content/architecture/m3query/blocks.md index 474afb51e9..92cbeece79 100644 --- a/site/content/m3query/architecture/blocks.md +++ b/site/content/architecture/m3query/blocks.md @@ -1,6 +1,6 @@ --- title: "Blocks" -weight: 1 +weight: 2 --- **Please note:** This documentation is a work in progress and more detail is required. diff --git a/site/content/m3query/architecture/fanout.md b/site/content/architecture/m3query/fanout.md similarity index 100% rename from site/content/m3query/architecture/fanout.md rename to site/content/architecture/m3query/fanout.md diff --git a/site/content/m3query/architecture/functions.md b/site/content/architecture/m3query/functions.md similarity index 100% rename from site/content/m3query/architecture/functions.md rename to site/content/architecture/m3query/functions.md diff --git a/site/content/architecture/m3query/overview.md b/site/content/architecture/m3query/overview.md new file mode 100644 index 0000000000..4c07967bf9 --- /dev/null +++ b/site/content/architecture/m3query/overview.md @@ -0,0 +1,19 @@ +--- +title: "Overview" +weight: 1 +--- + +M3 Query and M3 Coordinator are written entirely in Go, M3 Query is as a query engine for [M3DB](https://m3db.io/) and M3 Coordinator is a remote read/write endpoint for Prometheus and M3DB. To learn more about Prometheus's remote endpoints and storage, [see here](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage). + +M3 Query is a service that exposes all metrics query endpoints along with +metrics time series metadata APIs that return dimensions and labels of metrics +that reside in a M3DB cluster. + +**Note**: M3 Coordinator, and by proxy M3DB, by default includes the M3 +Query endpoints accessible on port 7201. +For production deployments it is recommended to deploy it as a +dedicated service to ensure you can scale the memory heavy query role separately +from the metrics ingestion write path of writes through M3 Coordinator to M3DB +database role nodes. This allows excessive queries to primarily affect the +dedicated M3 Query service instead of interrupting service to the write +ingestion pipeline. diff --git a/site/content/faqs/faqs.md b/site/content/faqs/faqs.md index 83ec62792a..b76a6c3c19 100644 --- a/site/content/faqs/faqs.md +++ b/site/content/faqs/faqs.md @@ -33,7 +33,7 @@ If you’re adding namespaces, the m3dbnode process will pickup the new namespac If you’re removing or modifying an existing namespace, you’ll need to restart the m3dbnode process in order to complete the namespace deletion/modification process. It is recommended to restart one node at a time and wait for a node to be completely bootstrapped before restarting another node. - **How do I set up aggregation in the coordinator?** -Refer to the [Aggregation section](/docs/how_to/query) of the M3Query how-to guide. +Refer to the [Aggregation section](/docs/how_to/m3query) of the M3Query how-to guide. - **How do I set up aggregation using a separate aggregation tier?** See this [WIP documentation](https://github.com/m3db/m3/pull/1741/files#diff-0a1009f86783ca8fd4499418e556c6f5). diff --git a/site/content/how_to/_index.md b/site/content/how_to/_index.md index af7e951e12..dc1fc4a72d 100644 --- a/site/content/how_to/_index.md +++ b/site/content/how_to/_index.md @@ -1,6 +1,4 @@ +++ title = "How To Guides" -date = 2020-04-01T19:26:56-04:00 -weight = 6 -chapter = true +weight = 7 +++ \ No newline at end of file diff --git a/site/content/how_to/aggregator.md b/site/content/how_to/m3aggregator.md similarity index 94% rename from site/content/how_to/aggregator.md rename to site/content/how_to/m3aggregator.md index 807ec21679..cc642bf592 100644 --- a/site/content/how_to/aggregator.md +++ b/site/content/how_to/m3aggregator.md @@ -1,14 +1,9 @@ --- -title: Setting up M3 Aggregator -menuTitle: M3Aggregator +title: Aggregate Metrics with M3 Aggregator weight: 5 --- -## Introduction - -`m3aggregator` is used to cluster stateful downsampling and rollup of metrics before they are store in M3DB. The M3 Coordinator also performs this role but is not cluster aware. This means metrics will not get aggregated properly if you send metrics in round robin fashion to multiple M3 Coordinators for the same metrics ingestion source (e.g. Prometheus server). - -Similar to M3DB, `m3aggregator` supports clustering and replication by default. This means that metrics are correctly routed to the instance(s) responsible for aggregating each metric and multiple `m3aggregator` replicas can be configured such that there are no single points of failure for aggregation. +{{< fileinclude file="m3aggregator_intro.md" >}} ## Configuration diff --git a/site/content/how_to/query.md b/site/content/how_to/m3query.md similarity index 76% rename from site/content/how_to/query.md rename to site/content/how_to/m3query.md index 41f843feca..37caa34295 100644 --- a/site/content/how_to/query.md +++ b/site/content/how_to/m3query.md @@ -1,35 +1,56 @@ --- -title: Setting up M3 Query -menuTitle: M3Query +title: Query Data with M3 Query weight: 4 --- -M3 Query is used to query data that is stored in M3DB. For instance, if you are using the Prometheus remote write endpoint with [m3coordinator](/docs/integrations/prometheus), you can use m3query instead of the Prometheus remote read endpoint. By doing so, you get all of the benefits of m3query's engine such as [block processing](/docs/m3query/architecture/blocks/). Furthermore, since m3query provides a Prometheus compatible API, you can use 3rd party graphing and alerting solutions like Grafana. +{{< fileinclude file="m3query_intro.md" >}} -## Configuration +## Prerequisites -Before setting up m3query, make sure that you have at least [one M3DB node running](/docs/quickstart). In order to start m3query, you need to configure a `yaml` file, that will be used to connect to M3DB. Here is a link to a [sample config](https://github.com/m3db/m3/blob/master/src/query/config/m3query-local-etcd.yml) file that is used for an embedded etcd cluster within M3DB. +Before running M3 Query, you need to have at least [one M3DB node running](/docs/quickstart). -### Running +## Configuring M3 Query -You can run m3query by either building and running the binary yourself: +To start M3 Query, you need a YAML configuration file that specifies the M3 nodes it connects to. + +{{% notice tip %}} +You can use [this sample configuration file](https://github.com/m3db/m3/blob/master/src/query/config/m3query-local-etcd.yml) for experimenting with a local cluster that uses an embedded etcd cluster. [Read the etcd documentation](/docs/operational_guide/etcd) for more details on running in production with an external etcd cluster. +{{% /notice %}} + +## Running M3 Query + +{{< tabs name="run_m3query" >}} +{{% tab name="Prebuilt Binaries" %}} + +M3 Query has pre-built binaries available for Linux and macOS. [Download the latest release from GitHub](https://github.com/m3db/m3/releases/latest). + +[Download the sample configuration file](https://github.com/m3db/m3/blob/master/src/query/config/m3query-local-etcd.yml) mentioned above. ```shell -make m3query -./bin/m3query -f ./src/query/config/m3query-local-etcd.yml +./bin/m3query -f .m3query-local-etcd.yml ``` -Or you can run it with Docker using the Docker file located at `$GOPATH/src/github.com/m3db/m3/docker/m3query/Dockerfile`. +{{% /tab %}} +{{% tab name="Self-built binary" %}} -### Namespaces +{{< fileinclude file="build_prerequisites.md" >}} -All namespaces that you wish to query from must be configured when [setting up M3DB](/docs/quickstart). If you wish to add or change an existing namespace, please follow the namespace operational guide [here](/docs/operational_guide/namespace_configuration). +Build and run the binary: -### etcd +```shell +make m3query +./bin/m3query -f ./src/query/config/m3query-local-etcd.yml +``` -The configuration file linked above uses an embedded etcd cluster, which is fine for development purposes. However, if you wish to use this in production, you will want an [external etcd](/docs/operational_guide/etcd) cluster. +{{% /tab %}} +{{% tab name="Docker" %}} - +```shell +docker run quay.io/m3db/m3query:v1.1.0 +``` + +{{% /tab %}} +{{< /tabs >}} ## Aggregation @@ -69,7 +90,6 @@ curl -X POST