maillog-mongodb: Add revised maillog-mongodb article

Prior article is https://www.fluentd.org/guides/recipes/maillog-mongodb.
Related to fluent#566.

Signed-off-by: Hiroshi Hatake <[email protected]>

docs/v1.0/maillog-mongodb.txt

@@ -0,0 +1,121 @@
+# Parse Postfix Maillogs and Store Them in MongoDB
+
+It is helpful to have maillogs stored in a semi-structured manner for future auditing and root cause analysis. This brief solution guide shows you how to parse Postfix maillogs and store them into MongoDB in near real-time.
+
+## Prerequisites
+
+- A basic understanding of Fluentd
+- A Postfix MTA running locally
+
+In this guide, we assume we are running [td-agent](https://www.fluentd.org/download) on Ubuntu Xenial.
+
+## Tailing the Postfix Maillog
+
+The first step is to set up [the tail input](in_tail) to tail the maillog.
+
+The Postfix maillog looks like this:
+
+```
+14-03-26T19:49:56+09:00 worker001 postfix/smtp[13747]: 31C5C1C000C: to=<[email protected]>,
+relay=mx.example.com[127.0.0.1]:25, delay=0.74, delays=0.06/0.01/0.25/0.42, dsn=2.0.0, status=sent (250 ok dirdel)
+```
+
+Which can be parsed with the following regular expression:
+
+```
+/^(?<date>[^ ]+) (?<host>[^ ]+) (?<process>[^:]+): (?<message>((?<key>[^ :]+)[ :])? ?((to|from)=<(?<address>[^>]+)>)?.*)$/
+```
+
+Thus, assuming the maillog is located at `/var/log/maillog`, to tail and parse the maillog, add the following to the Fluentd configuration (which, for `td-agent` is at `/etc/td-agent/td-agent.conf`).
+
+```
+<source>
+  @type tail
+  path /var/log/maillog
+  tag maillog.hostname_1
+  format /^(?<date>[^ ]+) (?<host>[^ ]+) (?<process>[^:]+): (?<message>((?<key>[^ :]+)[ :])? ?((to|from)=<(?<address>[^>]+)>)?.*)$/
+</source>
+```
+
+## Outputting to MongoDB
+
+The output plugin for MongoDB is bundled for `td-agent`. If you are running a vanilla Fluentd instance, run `gem install fluent-plugin-mongo` first.
+
+Add the following lines to output data into MongoDB:
+
+```
+<match maillog.*>
+  @type copy
+  <store>
+    # for debug (see /var/log/td-agent.log)
+    @type stdout
+  </store>
+  <store>
+    @type mongo
+    database fluentd #DB name
+    collection ${tag}
+    host YOUR_MONGODB_HOST
+    port YOUR_MONGODB_PORT
+    # ssl true
+    # user USER
+    # password PASSWORD
+  </store>
+</match>
+```
+
+The `collection ${tag}` parameter allows Fluentd to create a collection per tag. For example, if an event comes with tag = maillog.host1, it creates a collection named `maillog.host1`, and if another event comes with tag = maillog.host2, it creates a collection named `maillog.host2`. If you want to collect all maillogs into a single collection, use the following configuration instead.
+
+```
+<match maillog.*>
+  @type copy
+  <store>
+    # for debug (see /var/log/td-agent.log)
+    @type stdout
+  </store>
+  <store>
+    @type mongo
+    database fluentd #DB name
+    collection maillog #Collection name
+    host YOUR_MONGODB_HOST
+    port YOUR_MONGODB_PORT
+    <buffer>
+      @type memory
+      flush_interval 10s # for testing
+    </buffer>
+    # ssl true
+    # user USER
+    # password PASSWORD
+  </store>
+</match>
+```
+
+Also, notice that the MongoDB output supports SSL and password authentication (commented out).
+
+## Restart and Confirm That Data Flow into MongoDB
+
+Restart td-agent with `sudo service td-agent restart`. Then, run `tail` against /var/log/td-agent.log. You should see the following lines:
+
+```
+2012-07-12 15:19:03 +0000 maillog.hostname1: {"address":"[email protected]", "date":"2012-03-26T19:49:56+09:00", "host":"worker001", "key":"31C5C1C000C", "message":"31C5C1C000C: to=<[email protected]>, relay=mx.example.com[127.0.0.1]:25, delay=0.74, delays=0.06/0.01/0.25/0.42, dsn=2.0.0, status=sent (250 ok dirdel)", "process":"postfix/smtp[13747]"}
+```
+
+Now, go into MongoDB shell and check that the data is indeed imported.
+
+## What's Next?
+
+In production, you might want to remove writing output into stdout. So, use the following output configuration:
+
+```
+<match haproxy.*>
+  @type mongo
+  database fluentd #DB name
+  collection maillog #Collection name
+  host YOUR_MONGODB_HOST
+  port YOUR_MONGODB_PORT
+  # ssl true
+  # user USER
+  # password PASSWORD
+  </match>
+```
+
+Do you wish to store maillogs into other systems? Check out other [data outputs!](https://www.fluentd.org/dataoutputs).

lib/toc.en.v1.0.rb

@@ -47,6 +47,7 @@
     article 'free-alternative-to-splunk-by-fluentd', 'Free Alternative to Splunk by Fluentd + Elasticsearch', ['Splunk', 'Free Alternative']
     article 'splunk-like-grep-and-alert-email', 'Email Alerts like Splunk', ['Splunk', 'Alerting']
     article 'parse-syslog', 'Parse Syslog Messages Robustly'
+    article 'maillog-mongodb', 'Parse Postfix Maillogs and Store Them in MongoDB'
   end
   category 'data-analytics', 'Data Analytics' do
     article 'http-to-td', 'Data Analytics with Treasure Data', ['Treasure Data', 'Hadoop', 'Hive']