Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update training from Kibana to Loki #219

Draft
wants to merge 14 commits into
base: master
Choose a base branch
from
5 changes: 3 additions & 2 deletions .vscode/extensions.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"recommendations": [
"streetsidesoftware.code-spell-checker"
"streetsidesoftware.code-spell-checker",
"DavidAnson.vscode-markdownlint"
]
}
}
12 changes: 10 additions & 2 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,13 @@
"openshift",
"OCIO",
"parkade"
]
}
],
"files.eol": "\n",
"files.insertFinalNewline": true,
"markdownlint.config": {

"MD033": {
"allowed_elements": ["kbd"]
}
}
}
62 changes: 9 additions & 53 deletions 101-lab/content/12_logging_and_visualizations.md
Original file line number Diff line number Diff line change
@@ -1,65 +1,21 @@
# Logging and Visualizations

<kbd>[![Video Walkthrough Thumbnail](././images/12_logging_thumb.png)](https://youtu.be/zDAJcN5yTCg)</kbd>
<kbd>[![Video Walkthrough Thumbnail](././images/12_logging_thumb.png)](TBD)</kbd>

### EFK for Aggregated Logs
The OpenShift platform provides an aggregated logging stack that is automatically configured to centralize and store logs from application pods. These logs are only retained for a short period of time, currently about 14 days, but can be used to help identify issues with application pods.
## Loki for Aggregated Logs

Kibana is the primary interface for viewing and querying logs.
The OpenShift platform provides an aggregated logging stack that is automatically configured to centralize and store logs from application pods. These logs are only retained for a short period of time, currently about 14 days, but can be used to help identify issues with application pods.

#### Access the archive link from a pod
The shortcut towards accessing the Kibana is from the `Logs` tab of a running pod. Kibana can also be accessed directly at its [url](https://kibana-openshift-logging.apps.silver.devops.gov.bc.ca/).
Loki is the primary interface for viewing and querying logs.

### Access the archive link from a pod

- Select the running `rocketchat-[username]` pod and select the Logs tab
You can access Loki in the OpenShift console in the Developer mode under Observe -> Logs.

<kbd>![](./images/10_logging_01.png)</kbd>
<kbd>![12_logging_01](./images/12_logging_01.png)</kbd>

- Click on the "Show in Kibana" link to go to Kibana
- Kibana login is setup with SSO, you will see the same login page as of OpenShift console
- Kibana inherits the same RBAC as OpenShift, you will only have access to the same namespace/project set as from OpenShift cluster
Or, select the running `rocketchat-[username]` pod and select the Aggregated Logs tab

- Once you have logged in, for the first time you be asked to setup a search index. Follow the 2 steps from Kibana with the following value:
- Index pattern: `app*`
- Timestamp field name: `@timestamp`

<kbd>![](./images/10_logging_setup_01.png)</kbd>

<kbd>![](./images/10_logging_setup_02.png)</kbd>


- Click the 'discover' tab and review the logging interface and the query that has been automatically populated (there are more examples to explore at the end of this section)

<kbd>![](./images/10_logging_02.png)</kbd>

- Modify the query and time picker to select the entire namespace within the last few hours. First, I'm going to edit the query `kubernetes.namespace_name:"[-dev]"`, but replacing `[-dev]` with the name of the namespace I'm using. In the example, this is `kubernetes.namespace_name:"d8f105-dev"`. Next, I'll add a filter based on the `@timestamp` of the log messages, checking if each log entry `is between` the time periods `now-3h` and `now` and only displaying those logs.

<kbd>![](./images/10_logging_03.png)</kbd>

<kbd>![](./images/12_kibana_filter.png)</kbd>

- To see quick summary charts of a particular field, click the field name in the left menu of selected or available fields. In this case, let's click on the `kubernetes.pod_name` field.

<kbd>![](./images/12_kibana_timestamp.png)</kbd>

- Let's visualize which pods have been generating the most logs in the last 15 minutes. Click the 'visualize' button at the bottom of our `kubernetes.pod_name` field. Then, in the top right hand corner of the Kibana interface, look for the option to change the time range. Change this to `Last 15 minutes` from the `Quick` menu. You may wish to explore experimenting with visualizing other fields and time ranges.

<kbd>![](./images/12_kibana_time_range.png)</kbd>

<kbd>![](./images/12_kibana_visualization.png)</kbd>

#### Some useful queries you can try on:
- Get logs for the whole namespace:
```sql
kubernetes.namespace_name:"[-dev]"
```
- Use application labels to query logs from the same deployment:
```sql
kubernetes.namespace_name:"[-dev]" AND kubernetes.flat_labels:"deployment=[deployment_name]"
```
- Get error logs only:
```sql
kubernetes.namespace_name:"[-dev]" AND level:error
```
<kbd>![12_logging_02](./images/12_logging_02.png)</kbd>

Next page - [Metrics](./13_metrics.md)
6 changes: 3 additions & 3 deletions 101-lab/content/15_pod_lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ __Objective__: Create an init container
> If you wish to perform this from the cli with the `oc` tool, type `oc edit deployment/rocketchat-[username]`


<kbd>![](./images/12_pod_lifecycle_01.png)</kbd>
<kbd>![](./images/15_pod_lifecycle_01.png)</kbd>

- After replacing the URL below with the webhook URL from the previous step, add the following section of code under `spec: -> template: -> spec:`. As always with YAML, pay close attention to the formatting and indenting.

Expand Down Expand Up @@ -73,7 +73,7 @@ initContainers:

- Explore the `Pod Details` to notice the difference with the Init Container

<kbd>![](./images/12_pod_lifecycle_02.png)</kbd>
<kbd>![](./images/15_pod_lifecycle_02.png)</kbd>

In order to obtain logs from the init container, the `oc` command can be used by specifying `-c init`:

Expand Down Expand Up @@ -125,4 +125,4 @@ Be sure to save the changes to your YAML. Then:
## References
- https://blog.openshift.com/kubernetes-pods-life/

Next page - [Templates](./16_templates.md)
Next page - [Templates](./16_templates.md)
Binary file removed 101-lab/content/images/10_logging_01.png
Binary file not shown.
Binary file removed 101-lab/content/images/10_logging_02.png
Binary file not shown.
Binary file removed 101-lab/content/images/10_logging_03.png
Binary file not shown.
Binary file removed 101-lab/content/images/10_logging_04.png
Binary file not shown.
Binary file removed 101-lab/content/images/10_logging_setup_01.png
Binary file not shown.
Binary file removed 101-lab/content/images/10_logging_setup_02.png
Binary file not shown.
Binary file removed 101-lab/content/images/10_logging_viz_01.png
Binary file not shown.
Binary file removed 101-lab/content/images/11_logging_thumb.png
Binary file not shown.
Binary file removed 101-lab/content/images/12_kibana_filter.png
Binary file not shown.
Binary file removed 101-lab/content/images/12_kibana_time_range.png
Binary file not shown.
Binary file removed 101-lab/content/images/12_kibana_timestamp.png
Binary file not shown.
Binary file removed 101-lab/content/images/12_kibana_visualization.png
Binary file not shown.
Binary file added 101-lab/content/images/12_logging_01.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added 101-lab/content/images/12_logging_02.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion openshift-201/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The Openshift 201 Lab is divided into the following topics:
* [Java Application (Maven)](./pipelines.md)
* [Resource Management](./resource-mgmt.md)
* [Network Policy & ACS](./network-policy.md)
* [Application Logging with Kibana](./logging.md)
* [Application Logging with Loki](./logging.md)
* [Best Practices for Image Management](./image-management.md)
* [Pod Auto Scaling](./rh201-pod-auto-scale.md)
* [Post Outage Checkup](./post-outage-checkup.md)
Binary file removed openshift-201/images/logging/kibana-add-filter.png
Binary file not shown.
Binary file removed openshift-201/images/logging/kibana-discover.png
Binary file not shown.
Binary file removed openshift-201/images/logging/kibana-filter.png
Binary file not shown.
Binary file removed openshift-201/images/logging/kibana-main.png
Binary file not shown.
Binary file not shown.
Binary file removed openshift-201/images/logging/kibana-search-10.png
Binary file not shown.
Binary file removed openshift-201/images/logging/kibana-search.png
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added openshift-201/images/logging/loki-alert-1.png
Binary file added openshift-201/images/logging/loki-alert-2.png
Binary file added openshift-201/images/logging/loki-histogram.png
Binary file added openshift-201/images/logging/loki-logs-1.png
Binary file added openshift-201/images/logging/loki-main.png
Binary file added openshift-201/images/logging/loki-resources.png
Binary file modified openshift-201/images/logging/pod-logs-02.png
4 changes: 2 additions & 2 deletions openshift-201/instructor-resources/201-workshop-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,9 +174,9 @@ Tekton: Pipeline is kind of like a template, the pipeline run takes the paramete

### Recovery Checklist

## Application Logging with Kibana
## Application Logging with Loki

### Kibana Logging
### Loki Logging
### Usage

## Network Policy and ACS
Expand Down
Loading