updated readme

robcowart · May 31, 2019 · 738ad20 · 738ad20
1 parent cd52130
commit 738ad20
Showing 1 changed file with 118 additions and 33 deletions.
diff --git a/README.md b/README.md
@@ -1,51 +1,70 @@
-# sýnesis&trade; Lite for Suricata [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.me/robcowart)
+# sýnesis&trade; Lite for Suricata
+
+[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.me/robcowart)
+
 sýnesis&trade; Lite for Suricata provides basic log analytics for Suricata IDS/IPS using the Elastic Stack. It is a solution for the collection and analysis of Suricata "eve" JSON logs. This includes alerts, flows, http, dns, statistics and other log types.
 
-<img width="1163" alt="synesis_lite_suricata" src="https://user-images.githubusercontent.com/10326954/40805451-99e3f6e6-651e-11e8-9559-8e4535cd9b2c.png">
+![synesis_lite_suricata](https://user-images.githubusercontent.com/10326954/58703990-c851e500-83aa-11e9-988a-2c0dea8f4cc4.png)
 
 # Getting Started
+
 sýnesis&trade; Lite for Suricata is built using the Elastic Stack, including Elasticsearch, Logstash and Kibana. To install and configure sýnesis&trade; Lite for Suricata, you must first have a working Elastic Stack environment. The latest release requires Elastic Stack version 6.2 or later.
 
 Refer to the following compatibility chart to choose a release of sýnesis&trade; Lite for Suricata that is compatible with the version of the Elastic Stack you are using.
 
-Elastic Stack |  v1.x
+Elastic Stack | Release
 :---:|:---:
-6.2 | &#10003;
+7.1.x | &#10003; (v1.1.0)
+7.0.x | &#10003; (v1.1.0)
+6.x | &#10003; (v1.0.1)
 
 ## Suricata
-An example configuration for the Suricata eve output is provided in `suricata/suricata.yml`.
+
+Suggested configurations for the Suricata eve log output, and app_layer protocols is provided in the files found in the `suricata` directory.
 
 ## Setting up Elasticsearch
-Currently there is no specific configuration required for Elasticsearch. As long as Kibana and Logstash can talk to your Elasticsearch cluster you should be ready to go. The index template required by Elasticsearch will be uploaded by Logstash.
+
+Previous versions of the Elastic Stack required no special configuration for Elasticsearch. However changes made to Elasticsearch 7.x, require that the following settings be made in `elasticsearch.yml`:
+
+```text
+indices.query.bool.max_clause_count: 8192
+search.max_buckets: 100000
+```
 
 At high ingest rates (>5K logs/s), or for data redundancy and high availability, a multi-node cluster is recommended.
 
 ## Filebeat
-As Suricata is usually run on one or more Linux servers, the solution includes both [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) and [Logstash](https://www.elastic.co/guide/en/logstash/current/introduction.html). Filebeat is used to collect the log data on the system where Suricata is running, and ships it to Logstash via the Beats input. An example Filebeat prospector configuration is included in `filebeat/filebeat.yml`.
+
+As Suricata is usually run on one or more Linux servers, the solution includes both [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html) and [Logstash](https://www.elastic.co/guide/en/logstash/current/introduction.html). Filebeat is used to collect the log data on the system where Suricata is running, and ships it to Logstash via the [beats](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html) input. An example Filebeat log input configuration is included in `filebeat/filebeat.yml`.
 
 ## Setting up Logstash
+
 The sýnesis&trade; Lite for Suricata Logstash pipeline is the heart of the solution. It is here that the raw flow data is collected, decoded, parsed, formatted and enriched. It is this processing that makes possible the analytics options provided by the Kibana [dashboards](#dashboards).
 
 Follow these steps to ensure that Logstash and sýnesis&trade; Lite for Suricata are optimally configured to meet your needs. 
 
 ### 1. Set JVM heap size.
+
 To increase performance, sýnesis&trade; Lite for Suricata takes advantage of the caching and queueing features available in many of the Logstash plugins. These features increase the consumption of the JVM heap. Additionally the size of the IP reputation dictionary `ip_rep_basic.yml` can also increase heap usage. The JVM heap space used by Logstash is configured in `jvm.options`. It is recommended that Logstash be given 4GB of JVM heap. This is configured in `jvm.options` as follows:
 
-```
+```text
 -Xms4g
 -Xmx4g
 ```
 
 ### 2. Add and Update Required Logstash plugins
-It is also recommended that you always use the latest version of the [DNS](https://www.elastic.co/guide/en/logstash/current/plugins-filters-dns.html) filter. This can achieved by running the following command:
 
-```
+Ensure that all Logstash plugins are up to date by executing the following command.
+
+```shell
 LS_HOME/bin/logstash-plugin update logstash-filter-dns
 ```
 
 ### 3. Copy the pipeline files to the Logstash configuration path.
+
 There are four sets of configuration files provided within the `logstash/synlite_suricata` folder:
-```
+
+```text
 logstash
   `- synlite_suricata
        |- conf.d  (contains the logstash pipeline)
@@ -63,6 +82,7 @@ SYNLITE_SURICATA_TEMPLATE_PATH | The path to where index templates are located |
 SYNLITE_SURICATA_GEOIP_DB_PATH | The path where the GeoIP DBs are located | /etc/logstash/synlite_suricata/geoipdbs
 
 ### 4. Setup environment variable helper files
+
 Rather than directly editing the pipeline configuration files for your environment, environment variables are used to provide a single location for most configuration options. These environment variables will be referred to in the remaining instructions. A [reference](#environment-variable-reference) of all environment variables can be found [here](#environment-variable-reference).
 
 Depending on your environment there may be many ways to define environment variables. The files `profile.d/synlite_suricata.sh` and `logstash.service.d/synlite_suricata.conf` are provided to help you with this setup.
@@ -72,16 +92,18 @@ Recent versions of both RedHat/CentOS and Ubuntu use systemd to start background
 > Remember that for your changes to take effect, you must issue the command `sudo systemctl daemon-reload`.
 
 ### 5. Add the sýnesis&trade; Lite for Suricata pipeline to pipelines.yml
+
 Logstash 6.0 introduced the ability to run multiple pipelines from a single Logstash instance. The `pipelines.yml` file is where these pipelines are configured. While a single pipeline can be specified directly in `logstash.yml`, it is a good practice to use `pipelines.yml` for consistency across environments.
 
 Edit `pipelines.yml` (usually located at `/etc/logstash/pipelines.yml`) and add the sýnesis&trade; Lite for Suricata pipeline (adjust the path as necessary).
 
-```
+```text
 - pipeline.id: synlite_suricata
   path.config: "/etc/logstash/synlite_suricata/conf.d/*.conf"
 ```
 
 ### 6. Configure inputs
+
 By default Filebeat data will be recieved on all IPv4 addresses of the Logstash host using the default TCP port 5044. You can change both the IP and port used by modifying the following environment variables:
 
 Environment Variable | Description | Default Value
@@ -90,6 +112,7 @@ SYNLITE_SURICATA_BEATS_HOST | The IP address on which to listen for Filebeat mes
 SYNLITE_SURICATA_BEATS_PORT | The TCP port on which to listen for Filebeat messages | 5044
 
 ### 7. Configure Elasticsearch output
+
 Obviously the data needs to land in Elasticsearch, so you need to tell Logstash where to send it. This is done by setting these environment variables:
 
 Environment Variable | Description | Default Value
@@ -101,6 +124,7 @@ SYNLITE_SURICATA_ES_PASSWD | The username for the connection to Elasticsearch |
 > If you are only using the open-source version of Elasticsearch, it will ignore the username and password. In that case just leave the defaults.
 
 ### 8. Enable DNS name resolution (optional)
+
 In the past it was recommended to avoid DNS queries as the latency costs of such lookups had a devastating effect on throughput. While the Logstash DNS filter provides a caching mechanism, its use was not recommended. When the cache was enabled all lookups were performed synchronously. If a name server failed to respond, all other queries were stuck waiting until the query timed out. The end result was even worse performance.
 
 Fortunately these problems have been resolved. Release 3.0.8 of the DNS filter introduced an enhancement which caches timeouts as failures, in addition to normal NXDOMAIN responses. This was an important step as many domain owner intentionally setup their nameservers to ignore the reverse lookups needed to enrich flow data. In addition to this change, I submitted am enhancement which allows for concurrent queries when caching is enabled. The Logstash team approved this change, and it is included in 3.0.10 of the plugin.
@@ -121,23 +145,29 @@ SYNLITE_SURICATA_DNS_FAILED_CACHE_SIZE | The cache size for failed DNS queries |
 SYNLITE_SURICATA_DNS_FAILED_CACHE_TTL | The time in seconds failed DNS queries are cached | 3600
 
 ### 9. Start Logstash
+
 You should now be able to start Logstash and begin collecting network flow data. Assuming you are running a recent version of RedHat/CentOS or Ubuntu, and using systemd, complete these steps:
+
 1. Run `systemctl daemon-reload` to ensure any changes to the environment variables are recognized.
 2. Run `systemctl start logstash`
+
 > NOTICE! Make sure that you have already setup the Logstash init files by running `LS_HOME/bin/system-install`. If the init files have not been setup you will receive an error.
+
 To follow along as Logstash starts you can tail its log by running:
 
-```
+```text
 tail -f /var/log/logstash/logstash-plain.log
 ```
+
 Logstash takes a little time to start... BE PATIENT!
 
 Logstash setup is now complete. If you are receiving data from Filebeat, you should have both `suricata-` and `suricata_stats-` daily indices in Elasticsearch.
 
 ## Setting up Kibana
+
 An API (yet undocumented) is available to import and export Index Patterns. The JSON files which contains the Index Pattern configurations are `synlite_suricata.index_pattern.json` and `synlite_suricata_stats.index_pattern.json`. To setup the Index Patterns run the following commands:
 
-```
+```text
 curl -X POST -u USERNAME:PASSWORD http://KIBANASERVER:5601/api/saved_objects/index-pattern/suricata-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @/PATH/TO/synlite_suricata.index_pattern.json
 
 curl -X POST -u USERNAME:PASSWORD http://KIBANASERVER:5601/api/saved_objects/index-pattern/suricata_stats-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @/PATH/TO/synlite_suricata_stats.index_pattern.json
@@ -146,6 +176,7 @@ curl -X POST -u USERNAME:PASSWORD http://KIBANASERVER:5601/api/saved_objects/ind
 Finally the vizualizations and dashboards can be loaded into Kibana by importing the `synlite_suricata.dashboards.json` file from within the Kibana UI. This is done in the Kibana `Management` app under `Saved Objects`.
 
 ### Recommended Kibana Advanced Settings
+
 You may find that modifying a few of the Kibana advanced settings will produce a more user-friendly experience while using sýnesis&trade; Lite for Suricata . These settings are made in Kibana, under `Management -> Advanced Settings`.
 
 Advanced Setting | Value | Why make the change?
@@ -156,65 +187,117 @@ state:storeInSessionStorage | true | Kibana URLs can get pretty large. Especiall
 timepicker:quickRanges | [see below](#recommended-setting-for-timepicker:quickRanges) | The default options in the Time Picker are less than optimal, for most logging and monitoring use-cases. Fortunately Kibana no allows you to customize the time picker. Our recommended settings can be found [see below](#recommended-setting-for-timepicker:quickRanges).
 
 ## Dashboards
+
 The following dashboards are provided.
 
 > NOTE: The dashboards are optimized for a monitor resolution of 1920x1080.
 
 ### Alerts - Overview
-![suricata_alerts_overview](https://user-images.githubusercontent.com/10326954/41160782-f4eee61e-6b30-11e8-90dc-b90d85553959.png)
+
+![suricata_alerts_overview](https://user-images.githubusercontent.com/10326954/58702438-1d3f2c80-83a6-11e9-8bc0-9ad13fb8fad4.png)
 
 ### Alerts - Messages
-![suricata_alerts_messages](https://user-images.githubusercontent.com/10326954/41160783-f53fcc3c-6b30-11e8-8375-9f254184a540.png)
+
+![suricata_alerts_messages](https://user-images.githubusercontent.com/10326954/58702444-23350d80-83a6-11e9-87ab-d61392dfa4d6.png)
 
 ### Threats - Public Attackers
-![suricata_threat_pub_attack](https://user-images.githubusercontent.com/10326954/41160777-f3ff57a2-6b30-11e8-9ac0-a77ecf6c988c.png)
+
+![suricata_threat_pub_attack](https://user-images.githubusercontent.com/10326954/58702567-64c5b880-83a6-11e9-9e3f-d2c850bbcfeb.png)
 
 ### Threats - At-Risk Servers
-![suricata_threat_risk_servers](https://user-images.githubusercontent.com/10326954/41160723-dcd08e98-6b30-11e8-8793-5b2464dd0685.png)
+
+![suricata_threat_risk_servers](https://user-images.githubusercontent.com/10326954/58702573-67281280-83a6-11e9-8378-e8de1f4f5ad1.png)
 
 ### Threats - At-Risk Services
-![suricata_threat_risk_services](https://user-images.githubusercontent.com/10326954/41160725-dcfb5d3a-6b30-11e8-8b35-fd0044130662.png)
+
+![suricata_threat_risk_services](https://user-images.githubusercontent.com/10326954/58702578-6a230300-83a6-11e9-94b0-2319c8e4715e.png)
 
 ### Threats - High-Risk Clients
-![suricata_threat_risk_clients](https://user-images.githubusercontent.com/10326954/41160778-f43acd3c-6b30-11e8-8d42-5923b25db30a.png)
+
+![suricata_threat_risk_clients](https://user-images.githubusercontent.com/10326954/58702582-6c855d00-83a6-11e9-957b-1305cd0d6d79.png)
 
 ### Flows - Overview
-![suricata_flows_overview](https://user-images.githubusercontent.com/10326954/41160766-f2431eda-6b30-11e8-8cfd-652c5902fa15.png)
+
+![suricata_flows_overview](https://user-images.githubusercontent.com/10326954/58702448-262ffe00-83a6-11e9-9a47-92b68ca54727.png)
 
 ### Flows - Top Talkers
-![suricata_flows_talkers](https://user-images.githubusercontent.com/10326954/41160769-f2d7037a-6b30-11e8-9f0d-067ea6372296.png)
+
+![suricata_flows_talkers](https://user-images.githubusercontent.com/10326954/58702453-29c38500-83a6-11e9-929b-b913f89fca51.png)
 
 ### Flows - Top Services
-![suricata_flows_services](https://user-images.githubusercontent.com/10326954/41160768-f2a97982-6b30-11e8-84cc-527f52f822fd.png)
+
+![suricata_flows_services](https://user-images.githubusercontent.com/10326954/58702454-2cbe7580-83a6-11e9-84c2-dca23084e946.png)
 
 ### Flows - Sankey
-![suricata_flows_sankey](https://user-images.githubusercontent.com/10326954/41160767-f277a81c-6b30-11e8-8130-0625c70e53a7.png)
+
+![suricata_flows_sankey](https://user-images.githubusercontent.com/10326954/58702461-2fb96600-83a6-11e9-8055-c6b9dd6c267a.png)
 
 ### Flows - Geo IP
-![suricata_flows_geo](https://user-images.githubusercontent.com/10326954/41160764-f1e2fd16-6b30-11e8-8a92-a08d406b8d5e.png)
+
+![suricata_flows_geo](https://user-images.githubusercontent.com/10326954/58702468-334ced00-83a6-11e9-9980-4353d1198364.png)
 
 ### Flows - Messages
-![suricata_flows_messages](https://user-images.githubusercontent.com/10326954/41160765-f211f490-6b30-11e8-9f9b-189114be23bb.png)
+
+![suricata_flows_messages](https://user-images.githubusercontent.com/10326954/58702476-36e07400-83a6-11e9-84ba-8f870cd85c19.png)
 
 ### HTTP - Overview
-![suricata_http_overview](https://user-images.githubusercontent.com/10326954/41160772-f35fadf6-6b30-11e8-9002-950dcb9ffc74.png)
+
+![suricata_http_overview](https://user-images.githubusercontent.com/10326954/58702477-39db6480-83a6-11e9-9dc4-482658bec364.png)
 
 ### HTTP - Messages
-![suricata_http_messages](https://user-images.githubusercontent.com/10326954/41160770-f327274c-6b30-11e8-9c30-33a6a552878c.png)
+
+![suricata_http_messages](https://user-images.githubusercontent.com/10326954/58702484-3cd65500-83a6-11e9-9e48-7a37413ca2af.png)
 
 ### DNS - Overview
-![suricata_dns_overview](https://user-images.githubusercontent.com/10326954/41160779-f47b45ec-6b30-11e8-872c-0bd1f16f0991.png)
+
+![suricata_dns_overview](https://user-images.githubusercontent.com/10326954/58702492-3fd14580-83a6-11e9-994d-5fac29edec81.png)
 
 ### DNS - Messages
-![suricata_dns_messages](https://user-images.githubusercontent.com/10326954/41160781-f4b413b8-6b30-11e8-8e9b-1574873f99f2.png)
+
+![suricata_dns_messages](https://user-images.githubusercontent.com/10326954/58702495-42cc3600-83a6-11e9-9f66-a4a09600e904.png)
+
+### SSH - Overview
+
+![suricata_ssh_overview](https://user-images.githubusercontent.com/10326954/58702499-45c72680-83a6-11e9-884c-0ff2e998c764.png)
+
+### SSH - Messages
+
+![suricata_ssh_messages](https://user-images.githubusercontent.com/10326954/58702506-48c21700-83a6-11e9-88aa-eba7ad875e0c.png)
+
+### TLS - Overview
+
+![suricata_tls_overview](https://user-images.githubusercontent.com/10326954/58702510-4c559e00-83a6-11e9-97ab-3ec43bc398de.png)
+
+### TLS - Messages
+
+![suricata_tls_messages](https://user-images.githubusercontent.com/10326954/58702514-4f508e80-83a6-11e9-9c16-1f3d3d304690.png)
+
+### SMB - Overview
+
+![suricata_smb_overview](https://user-images.githubusercontent.com/10326954/58702531-524b7f00-83a6-11e9-905c-2635b504e1a2.png)
+
+### SMB - Messages
+
+![suricata_smb_messages](https://user-images.githubusercontent.com/10326954/58702544-55df0600-83a6-11e9-8743-4cfa83d2c57c.png)
+
+### NFS - Overview
+
+![suricata_nfs_overview](https://user-images.githubusercontent.com/10326954/58702548-59728d00-83a6-11e9-9540-46140f14ef48.png)
+
+### NFS - Messages
+
+![suricata_nfs_messages](https://user-images.githubusercontent.com/10326954/58702553-5c6d7d80-83a6-11e9-9ada-417a209e2887.png)
 
 ### Raw Logs
-![suricata_raw_logs](https://user-images.githubusercontent.com/10326954/41160775-f3963448-6b30-11e8-8d16-22cbe71b80e3.png)
+
+![suricata_raw_logs](https://user-images.githubusercontent.com/10326954/58702560-5ecfd780-83a6-11e9-80a3-c181046cbe87.png)
 
 ### Statistics
-![suricata_stats](https://user-images.githubusercontent.com/10326954/41160776-f3ceb138-6b30-11e8-8baa-189b8c7cdee5.png)
+
+![suricata_stats](https://user-images.githubusercontent.com/10326954/58702564-61cac800-83a6-11e9-90a8-c75e5f4efedc.png)
 
 # Environment Variable Reference
+
 The supported environment variables are:
 
 Environment Variable | Description | Default Value
@@ -239,11 +322,12 @@ SYNLITE_SURICATA_BEATS_HOST | The IP address on which to listen for Filebeat mes
 SYNLITE_SURICATA_BEATS_PORT | The TCP port on which to listen for Filebeat messages | 5044
 
 # Recommended Setting for timepicker:quickRanges
+
 I recommend configuring `timepicker:quickRanges` for the setting below. The result will look like this:
 
 ![screen shot 2018-05-17 at 19 57 03](https://user-images.githubusercontent.com/10326954/40195016-8d33cac4-5a0c-11e8-976f-cc6559e4439a.png)
 
-```
+```text
 [
   {
     "from": "now/d",
@@ -357,4 +441,5 @@ I recommend configuring `timepicker:quickRanges` for the setting below. The resu
 ```
 
 # Attribution
+
 This product includes GeoLite data created by MaxMind, available from (http://www.maxmind.com)