Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add comments relating memory sizing. #88

Open
wants to merge 4 commits into
base: develop
Choose a base branch
from
Open

Add comments relating memory sizing. #88

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
eabf7e6
Add comments relating memory sizing.
matthewvon Nov 14, 2022
cee5866
spelling error fixed
matthewvon Nov 14, 2022
6049bca
address PR comments
matthewvon Nov 14, 2022
48fa0f8
address PR comments
matthewvon Nov 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion charts/stardog/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,10 +53,47 @@ Configuration Parameters
| `busybox.image.tag` | The Docker image tag for busybox image (used as a part of Stardog initialization) |
| `busybox.image.username` | The Docker registry username for busybox image registry (used as a part of Stardog initialization) |
| `busybox.image.password` | The Docker registry password for the busybox image registry (used as a part of Stardog initialization) |
| `additionalStardogProperties` | Allow adding additional settings to stardog.properties file |
| `additionalStardogProperties` | Allows adding additional settings to stardog.properties file |

The default values are specified in `values.yaml` as well as the required values for the ZooKeeper chart.

Memory Parameters
-----------------

Memory settings are part of both javaArgs and resources within
values.yaml. The settings must be coordinated.

Stardog will aggressively use all memory allocated to it via the
javaArgs settings of -Xmx and -XX:MaxDirectMemorySize. The total
memory used by Stardog is the sum of -Xmx and -XX:MaxDirectMemorySize.

The resources settings of requests:memory and optional limits:memory
must account for the javaArgs settings. requests:memory should be at
least 1g larger than -Xms and -XX:MaxDirectMemorySize total. 2g larger is
preferred. If you chose to establish resourse:limits, limits:memory
must be at least 1g larger than -Xmx and -XX:MaxDirectMemorySize total. A
too small limits:memory setting will likely result in a system crash.

Network based disk storage
--------------------------

Network based disk storage is not recommended. This include NFS, SMB,
and EFS based storage systems. Stardog instead recommends setting the
storageClass to gp2 on AWS and leaving the default values of standard
for Azure and GCP.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On AWS, we recommend gp2 as the storageClass. On GKS and AKS standard (the default) is what we recommend. I would just add something like "we instead recommend setting the storageClass to gp2 on AWS and leaving the default values of standard for Azure and GCP."

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify from discussions outside of this PR. I think we should call out a bit more details that the storage classes may not always be named this depending on how they are configured. Instead we should tell people to use block volume storage classes for their infrastructure and depending on their use case they may want to explore differences between the types, e.g. gp2 vs io2 or standard vs premium. We can also link to the k8s docs on storage classes for further information: https://kubernetes.io/docs/concepts/storage/storage-classes/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To include the suggestions from Paul's comment, I would edit it to read something like:
Stardog instead recommends using block volume storage classes such as gp2 on AWS or standard on other platforms. Depending on use case, it is recommended to explore differences between the types, e.g. gp2 vs io2 or standard vs premium.

If network based disk is used, the following should be part of the
additionalStardogProperties setting:

storage.envoptions.use_mmap_writes = false

Stardog uses a technique call memory mapping for high speed writes of
new data and data optimizations. Memory mapping is not appropriate
for network files systems. Memory mapping is simulated at the server
causing increased operating system memory usage, slower write
throughput, and increased chance of error / data corruption. The
above setting disables the use of memory mapping.

Upgrades
--------

Expand Down
4 changes: 4 additions & 0 deletions charts/stardog/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ cluster:
terminationGracePeriodSeconds: 300

# Java args for Stardog server
# Shown are memory sizing parameters. Total potential memory is sum of
# -Xmx and -XX:MaxDirectMemorySize. In this sample, the total memory is 3g
javaArgs: "-Xmx2g -Xms2g -XX:MaxDirectMemorySize=1g"

# Additional arguments passed to Stardog at server start
Expand Down Expand Up @@ -81,6 +83,8 @@ antiAffinity: requiredDuringSchedulingIgnoredDuringExecution

# Stardog should have at least 2 CPUs. The CPU request value is used to set
# -XX:ActiveProcessorCount=N for the JVM.
# If limits:memory is activated, it must be at least 1g larger than the
# total of javaArg's -Xmx and -XX:MaxDirectMemorySize. 2g larger is recommended.
resources:
requests:
cpu: 2
Expand Down