Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OECC demo #3

Open
wants to merge 20 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/Kafka/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Kafka Consumer/Producer #

To run `./comsume.ch`
To run `./comsume.sh`

To run `./producer.sh`
149 changes: 149 additions & 0 deletions examples/OECC/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# OpenEdge Command Center #

edselg marked this conversation as resolved.
Show resolved Hide resolved
Demo scripts to setup an OpenEdge Command Center environment with support for OpenTelemetry metrics.

The scripts are an updated version of the scripts used for the following blog post:
* https://www.progress.com/blogs/using-opentelemetry-metrics-support-openedge-azure

In the new version, the scripts can be used on an Ubuntu VM running on various virtualization options such as AWS, Azure, VirtualBox or VMWare Fusion.

Scripts run commands to detect the IP address of the machine.

### Requirements
* Ubuntu Linux 22.04
* user with sudo access
* OpenEdge media:
* PROGRESS_OE_12.8.3_LNX_64.tar.gz
* PROGRESS_OECC_SERVER_1.3.0_LNX_64.tar.gz
edselg marked this conversation as resolved.
Show resolved Hide resolved
* PROGRESS_OECC_AGENT_1.3.0_LNX_64.bin
* Reponse file for OpenEdge installation:
* response_12.8.3.ini
* Products:
* OE Adv. Ent. RDBMS
* 4GL Development System
* Progress ProdAS for OE

**Notes:**
* The OpenEdge media for both OpenEdge and OpenEdge Command Center can be downloaded from the Progress OpenEdge 12.8.x section in the ESD site.
* You can use the `OpenEdge-Samples/examples/OECC/config/response_12.8.3.ini` file as a template and specify the control codes in the file.
* The demo scripts install the components using the media and the response .ini file.
* The demo scripts install OpenJDK to `/usr/lib/jvm/jdk` and OpenEdge to `/psc/dlc`.
* The demo scripts install Docker Engine on Ubuntu using the instructions at https://docs.docker.com/engine/install/ubuntu/.

### Components
The infrastructure uses multiple components. To simplify the deployment, it uses a single virtual machine with some components running at the OS level and others running as a container using Docker.

* Ubuntu Virtual Machine latest 22.04 (tested 22.04.4)
* OpenEdge Database 12.8.3
* PAS for OpenEdge 12.8.3
* OpenEdge Command Center agent 1.3.0
* Docker (latest, version tested: 26.1.4)
Copy link
Contributor

@DustinGrau-PSC DustinGrau-PSC Jun 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should a user install Docker for Ubuntu? Could we add a link to something like this page or embed some instructions? https://docs.docker.com/engine/install/ubuntu/

Asking as my Ubuntu installation came with Docker 20.0.5 but following those instructions got me up to version 27.0.2 so it may be worth stating that as a reference for those people unfamiliar with Docker.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling ./scripts/setup.sh would call ../common/scripts/install_docker.sh which performs the installation of Docker on Ubuntu.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was honestly not aware of that, and apparently did not look that closely at the setup.sh script to realize.

This is why I've been incredibly pedantic during the code review and this is exactly what has stopped me from proceeding further than I would have liked. There are a LOT of assumptions made about the skillset of the user, familiarity with the technology, and perhaps some details missed because you are writing as the subject matter expert. It's easy to miss things or take for granted what you already know. I would ask that you take another look at the opening of the main README file, specifically the Requirements and Components sections to add notes about where each should come from.

For example, for the OpenEdge media, state those should come from the ESD site, and where they might be found. For the components such as Docker, Grafana, etc. state that those are installed by the scripts if that's the case--otherwise, explain how to manually install them (apt-get, etc.) so users know they need to have them installed before proceeding.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention of the demo scripts is to provide something that reduces the complexity of the installation process of OpenEdge Command Center and also gives the user the option to try the OpenTelemetry metrics support.
The demo scripts encapsulate the instructions. They are an updated version of the scripts used for the following blog post:

I am hoping to write a new version of the blog post.

Please try running ./scripts/setup.sh.

Thanks.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As someone who has had little exposure to any of this, I'm now curious how much is OECC and how much is for OTel? Would this have been better to demonstrate as 2 separate examples? OECC should just be OECC, OTel just OTel. By mixing components it's difficult to know which is required or optional for their respective technology.

We already have a separate Kafka folder that demonstrates just that technology--why mix these concepts under OECC?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If feel that we can go either way on this.

The idea of having both together was because, at the time that the initial scripts were written and perhaps still today, OpenTelemetry is one of the most important features when using OpenEdge Command Center.
Related link: https://www.progress.com/openedge/features/command-center

It might be likely that users exploring OpenEdge Command Center (server) would also be interested on taking a look at the OpenTelemetry support.

A demo for OpenTelemetry only needs the configuration of OpenEdge Command Center agent.
Perhaps, we can have a simplified version of the demo scripts that only needs OpenEdge Command Center agent.
The interface to setup the demo would still be the same:

cd <demo-directory>
./scripts/setup.sh

Thanks.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I were a new looking at this folder as just "OECC" I would assume that these are all required and normal components. Having to install several extra tools to get OECC running without differentiation makes this feel more complicated than it should be.

You already have a "common" scripts folder, so why not make these subtopics which isolates learning?

  • Examples/OECC
    • OECC (Standalone)
    • OECC + OpenTelemetry

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where are you expecting these subtopics?
Are thinking on the being part of OpenEdge-Samples/examples/OECC?
Or are you think on separate demos?

The current README.md file has a "Creating a Dashboard in Grafana" as a separate section on the demo.

I can add an optional section to the README.md on "Using OpenTelemetry Support in OpenEdge" and leave the core of the README.md to be OpenEdge Command Center.

Please confirm.

Thanks.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where did this work originate? Do we have a ticket that explains the purpose of the work, acceptance criteria, etc? Or is this all just off-the-cuff? We should really be tracking the work and should have discussed what goes into these examples as a matter of agreeing on the content. The more I'm diving into this the more confused I am over why I would need Docker and all these other tools just to install something that already has installer files. The OECC itself would be on a central server/location while the agent would be installed on multiple client machines, so it's hard to visualize using this in a distributed manner.

If the need for Docker is so the instance can be easily encapsulated, then there should be a Docker container for the OECC (server) component while a separate container with a PASOE instance and the agent would be installed together. That way it could demonstrate how to set up each side of the solution. This could all be a single document that is the OECC example itself--nothing more. The addition of OTel, Grafana, etc. just seem extraneous and should be it's own topic for after you've successfully set up OECC to monitor a PAS instance. That way the user could deploy the things needed to each side--either the server side of the components or the agent-monitored instances.

* OpenTelemetry Collector 0.53.0
* Prometheus (latest, version tested: v2.53.0)
* Grafana (latest, version tested: 9.5.20)
* MongoDB 4.4-rc-focal
* OpenEdge Command Center server 1.3.0

The OpenEdge Database, PAS for OpenEdge and the OpenEdge Command Center agent are installed at the OS level.
The OpenTelemetry Collector, Prometheus, Grafana, MongoDB, and the OpenEdge Command Center server are deployed using Docker.

The following diagram shows the components use for OpenTelemetry support.
![Components](./images/diagram.png)

See related diagram at
https://www.progress.com/openedge/features/command-center

### Steps

#### Preparing the installation with the prerequisites
1. Create `/install` directory used for the installation files.
~~~
sudo mkdir -p /install
sudo chown $USER /install
~~~

2. Copy the OpenEdge media files and the response file to `/install`.

#### Setting up the environment

1. Clone the OpenEdge-Samples repo:
~~~
cd
git clone https://github.com/progress/OpenEdge-Samples.git
~~~

2. Run `setup.sh` in the `OECC/scripts` directory. Script uses `sudo` to perform administrator tasks.
~~~
cd ~/OpenEdge-Samples/examples/OECC
./scripts/setup.sh
~~~

#### Testing the environment

1. Test the configuration.
~~~
./scripts/test.sh
~~~

2. Access OpenEdge Command Center via a web browser:
* http://&lt;host-machine&gt;:8000
* Use admin/admin to login

3. Access the PAS for OpenEdge instance via a web browser:
* http://&lt;host-machine&gt;:8810
* http://&lt;host-machine&gt;:8810/web/customer.p

4. Access Prometheus via a web browser:
* http://&lt;host-machine&gt;:9090
* Use DB_ in the Expression field to query for metrics for the OpenEdge database
* Use PASOE_ in the Expression field to query for metrics for PAS for OpenEdge

5. Access Grafana via a web browser:
* http://&lt;host-machine&gt;:3000
* Use admin/admin to login

**Notes:**
* The setup script, adds `/etc/rc.local` to start the configuration on startup of the machine. You can also use `./scripts/start.sh` and `./scripts/stop.sh` to manually start and stop the configuration.

### Creating a Dashboard in Grafana

**Steps**

1. Click on DATA SOURCES then on Prometheus to create a data source with the following values:
* Prometheus server URL: http://prometheus:9090
* Prometheus is a reference to a service in the `docker-compose.yaml` file and can be used as a hostname.
* Scroll down and click on Save & Test

2. Click on Home then on DASHBOARDS to create a dashboard.
3. Click on Add visualization to add a visualization for an OpenEdge database metric.
* Select Prometheus as the data source.
* Click on "Select metric" and click on Open.

![Selecting a Metric #1](./images/selecting_a_metric1.png)

* Search for "DB_" and select the BufferHits metric from the list.

![Selecting a Metric #2](./images/selecting_a_metric2.png)

* Click on Run queries.
* Click on Time series to change the visualization to use a Gauge.
* Change the Panel Title to "Buffer Hits"
* Scroll down and change the Unit in "Standard options" to Misc / Pecent (0-100).
* Scroll down and click on the Red dot for the Threshold to use Green.
* Click on the Green dot for Base to use Orange.
* Click on Apply.

![Visualization](./images/bufferhits_visualization.png)

4. Click on Add / Visualization (on toolbar) to add a visualization for a PASOE metric.
* Click on "Select metric" and click on Open.
* Search for "getrequests" and select the getRequests metric from the list.
* Click on Run queries.
* Change the Panel Title to "Get Requests".
* Click on Apply.
5. Click on the "Save dashboard" button (floppy disk icon on toolbar) to save the dashboard.

![Dashboard](./images/openedge_dashboard.png)

Notes:
* Examples of Grafana dashboards can be found at https://grafana.com/grafana/dashboards/
45 changes: 45 additions & 0 deletions examples/OECC/config/customer.p
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/** ------------------------------------------------------------------------
File : customer.p
Purpose : Return records from the Customer table as JSON.
Syntax :
Description :
@author egarcia
Modified : Thu Jun 27 15:32:00 EST 2024
Notes :
* Deploy to openedge/ directory in PAS instance
using OpenEdge.Web.CompatibilityHandler
* Access using http://<host-machine>:8810/web/customer.p
---------------------------------------------------------------------- */
USING Progress.Json.ObjectModel.JsonObject.
edselg marked this conversation as resolved.
Show resolved Hide resolved

CREATE WIDGET-POOL.

{src/web2/wrap-cgi.i}

DEFINE TEMP-TABLE ttCustomer LIKE Customer.
DEFINE DATASET dsCustomer FOR ttCustomer.

RUN process-web-request.

PROCEDURE outputHeader :
output-content-type ("application/json":U).
END PROCEDURE.

PROCEDURE process-web-request :
DEFINE VARIABLE oJsonObject AS Progress.Json.ObjectModel.JsonObject NO-UNDO.
DEFINE VARIABLE lChar AS LONGCHAR NO-UNDO.
RUN outputHeader.

EMPTY TEMP-TABLE ttCustomer.
FOR EACH Customer WHERE Customer.CustNum < 3110 NO-LOCK:
CREATE ttCustomer.
BUFFER-COPY Customer TO ttCustomer.
END.

oJsonObject = NEW JsonObject().
oJsonObject:READ(DATASET dsCustomer:HANDLE).

lChar = oJsonObject:GetJsonText().
{&OUT-LONG} lChar.

END PROCEDURE.
9 changes: 9 additions & 0 deletions examples/OECC/config/oepas1.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[AppServer.SessMgr]
agentStartupParam=-T "${catalina.base}/temp" -db sports2020 -S 20000

[AppServer.Agent]
numInitialSessions=2

[oepas1.ROOT.WEB]
adapterEnabled=1
defaultHandler=OpenEdge.Web.CompatibilityHandler
19 changes: 19 additions & 0 deletions examples/OECC/config/otagentoedb.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
exporter:
name: "otlp"
endpoint: "http://192.168.56.215:4317"
protocol: "grpc"
connectionretry: 20
timeout: 10


oedbInstances:
- dbname: sports2020
host: localhost
port: 20000
user: dba
password: dba
metricsregex:
otherdbconnparams:
dbconnectionretry: 5
dbschedule: 30
dbduration: seconds
13 changes: 13 additions & 0 deletions examples/OECC/config/otagentpasoe.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
exporter:
name: "otlp"
endpoint: "http://192.168.56.215:4317"
protocol: "grpc"
connectionretry: 20
timeout: 10

pasInstances:
- pasdir: "/psc/wrk/oepas1"
passchedule: 30
pasconnectionretry: 5
pasduration: seconds
metricsregex:
11 changes: 11 additions & 0 deletions examples/OECC/config/rc.local
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#!/bin/sh -e
edselg marked this conversation as resolved.
Show resolved Hide resolved
#

export DLC=/psc/dlc
edselg marked this conversation as resolved.
Show resolved Hide resolved
export PATH=$DLC/bin:$PATH

cd /psc/wrk
proserve sports2020 -S 20000 -n 30
/psc/wrk/oepas1/bin/tcman.sh start

exit 0
Loading