diff --git a/docs/launchpad/advanced-tutorials/kubeadm-upgrade-cluster-config.md b/docs/launchpad/advanced-tutorials/kubeadm-upgrade-cluster-config.md index d605d06e..890b31a9 100644 --- a/docs/launchpad/advanced-tutorials/kubeadm-upgrade-cluster-config.md +++ b/docs/launchpad/advanced-tutorials/kubeadm-upgrade-cluster-config.md @@ -21,6 +21,9 @@ kubeadm upgrade plan ``` **3:** Perform the upgrade: + +*Note*: If you have local patches applied to your Kubernetes setup (ie. altering the kube-scheduler or kube-controller-manager configurations for better performance under specific workloads or hardware configurations), ensure they are included or updated appropriately during the upgrade process. To do this pass the `--patches /path/to/your/patches` flag to your `kubeadm upgrade apply` command. + ```bash kubeadm upgrade apply v1.28.3 ``` diff --git a/docs/launchpad/docs-map.md b/docs/launchpad/docs-map.md index 97bb9465..4da6c49e 100644 --- a/docs/launchpad/docs-map.md +++ b/docs/launchpad/docs-map.md @@ -4,7 +4,9 @@ sidebar_position: 1 # Launchpad Documentation -Everything you need to know about the Launchpad project. +Launchpad is a comprehensive toolkit designed for running a Graph Protocol Indexer on Kubernetes, aimed at providing the fastest route to production deployments of multi-chain indexing software stacks with robust security and performance defaults. Suitable for environments ranging from a single node cluster to large scale multi-region clusters. Launchpad is also comprised of an opinionated set of tools that run on your local machine, that are layered to offer a declarative workflow for managing your deployment stack. Key components of Launchpad include the Launchpad Starter ([`graphops/launchpad-starter`](https://github.com/graphops/launchpad-starter)), which serves as the initial setup point for new deployments; Launchpad Charts ([`graphops/launchpad-charts`](https://github.com/graphops/launchpad-charts)), a collection of Helm Charts for blockchains and web3 applications; and Launchpad Namespaces ([`graphops/launchpad-namespaces`](https://github.com/graphops/launchpad-namespaces)), which are preconfigured Kubernetes Namespaces that utilize Helmfile for enhanced management. + +Here's a guide to help you navigate this documentation based on the information you're seeking: ## First steps diff --git a/docs/launchpad/faq.md b/docs/launchpad/faq.md index 8aca5925..dadc534e 100644 --- a/docs/launchpad/faq.md +++ b/docs/launchpad/faq.md @@ -17,6 +17,8 @@ Here are answers to some commonly asked questions. If you have a question that i - [Is there a way to inject a pretuned postgres config into the chart?](#is-there-a-way-to-inject-a-pretuned-postgres-config-into-the-chart) - [Why are my stateful pods in `Pending` state and their expected `pvc` showing `WaitForFirstConsumer` errors?](#why-are-my-stateful-pods-in-pending-state-and-their-expected-pvc-showing-waitforfirstconsumer-errors) - [Do I need to use Cilium for Launchpad?](#do-i-need-to-use-cilium-for-launchpad) + - [How active is the Launchpad project?](#how-active-is-the-launchpad-project) + - [I'm not ready to use Launchpad but I use Kubernetes](#im-not-ready-to-use-launchpad-but-i-use-kubernetes) - [Need More Help?](#need-more-help) --- @@ -93,6 +95,21 @@ Normal WaitForFirstConsumer 6m52s persistentvolume-controlle It's important to acknowledge that while Cilium has better performance and features than Calico, it is a bit trickier to set up. Our decision isn't influenced by Launchpad; it's purely a preference based on the operational benefits that Cilium brings to our infrastructure. +--- + +### How active is the Launchpad project? + +**Q: How often is the Launchpad project updated?** + +**A:** The GraphOps team actively maintains the Launchpad project as it is integral to their indexing infrastructure. For details on how new versions of applications (ie. Erigon, Graph-node etc) are integrated into Launchpad Charts and Launchpad Namespaces, please refer to our [Release Channels documentation](release-channels.md). Additionally, you can learn about our criteria for supporting different Launchpad Namespaces by visiting the [Supported Namespaces page](supported-namespaces.md). These resources provide a comprehensive view of our update frequency and decision-making processes regarding the inclusion of new features and applications. + +--- + +### I'm not ready to use Launchpad but I use Kubernetes + +**Q: Is this project relevant to me if I use Kubernetes to manage blockchain infrastructure?** + +**A:** Absolutely, the Launchpad project is designed with modularity at its core, making it highly adaptable for users who aren't ready to fully implement all of its components. You can benefit from using our Launchpad Charts with Helm to manage specific components of your blockchain infrastructure independently. Additionally, our charts are compatible with GitOps workflows, allowing you to integrate them seamlessly into your existing management practices. For further insights into how you can leverage the modular aspects of our project, please visit our [Modularity documentation](modularity.md). ## Need More Help? diff --git a/docs/launchpad/intro.md b/docs/launchpad/intro.md index e1a38859..b6c05834 100644 --- a/docs/launchpad/intro.md +++ b/docs/launchpad/intro.md @@ -24,8 +24,9 @@ There are three major components to be aware of: Are you interested in exploring Launchpad but not ready to adopt the entire stack? Explore our [Modularity](modularity) page to discover how you can selectively integrate elements of Launchpad, like `launchpad-starter`, `launchpad-charts`, and `launchpad-namespaces`, to fit your specific needs without committing to a full end-to-end implementation. -## Next Steps +## Next steps +- Visit our [Documentation Map](docs-map) for an overview of where to find all the information you need - Read the [Prerequisites](prerequisites) section to understand what you need to started - Read the [Quick Start guide](quick-start) to get up and running - Look at the repositories above on GitHub to understand how they work \ No newline at end of file diff --git a/docs/launchpad/quick-start.md b/docs/launchpad/quick-start.md index 9571d7d0..f0d138a6 100644 --- a/docs/launchpad/quick-start.md +++ b/docs/launchpad/quick-start.md @@ -4,8 +4,19 @@ sidebar_position: 3 # Quick Start +We have designed Launchpad to be modular so that you can implement the whole project or parts of it as best suits your needs. Checkout [this page](modularity.md) for more info about the modularity of Launchpad. + Make sure you have all the [Prerequisites](prerequisites) before starting. +To start jump to the relevant section based on how you're using the project: + +- [Using Launchpad End to End](#using-launchpad-end-to-end) +- [Using Launchpad Charts and Helmfile](#using-helmfile-and-launchpad-charts) + +## Using Launchpad end to end + +This section takes you through steps of getting started using all aspects of the Launchpad project. + ### Install Taskfile Launchpad has a large number of tooling dependencies that will run on your local machine. The most important dependency is [Taskfile](https://taskfile.dev). @@ -329,4 +340,332 @@ Launchpad comes with a built in task to do this: ```shell task launchpad:pull-upstream-starter -``` \ No newline at end of file +``` + +## Using Helmfile and Launchpad Charts + +This guide will cover two primary ways to deploy blockchain-related resources in Kubernetes using Launchpad charts: deploying all components at once using Helmfile and deploying individual components directly using Helm charts. + +### Prerequisites + +Ensure you have [helm](https://github.com/helm/helm), [helmfile](https://github.com/helmfile/helmfile) and it's dependency [helm-diff](https://github.com/databus23/helm-diff) installed on your local machine. This guide assumes familiarity with basic Helm and Helmfile operations. + +Before proceeding with this guide, make sure the following tools are installed on your local machine: + +- [Helm](https://github.com/helm/helm): The package manager for Kubernetes, essential for managing and deploying applications. +- [Helmfile](https://github.com/helmfile/helmfile): A tool to help streamline the use of Helm charts, enabling better management of Helm chart configurations. +- [Helm-diff](https://github.com/databus23/helm-diff): A Helm plugin that helps visualize differences between your Helmfile configurations and what is actually deployed in your cluster. This plugin is a dependency for effectively using Helmfile. +- (Optional)[Kustomize](https://github.com/kubernetes-sigs/kustomize): A tool for customizing Kubernetes configurations beyond what is available with Helm, useful for more complex deployment scenarios. +This guide assumes you are familiar with basic operations of Helm and Helmfile. + +### Deploying using Launchpad-charts directly + +If you prefer to use individual components of Launchpad, such as Launchpad Charts, you can add the Launchpad Helm repository and install charts directly: + +```shell +helm repo add graphops https://graphops.github.io/launchpad-charts +helm install my-release graphops/ --values +``` + +#### Key Consideration + +Before proceeding, it is important to note that most Kubernetes clusters do not come pre-configured with a [Container Storage Interface (CSI)](https://kubernetes-csi.github.io/docs/) for handling storage volumes. This guide relies on the ability to create storage volumes. It is also necessary to have an Ingress controller installed and configured, as it is essential for managing traffic to and from your applications. + +### Deploying using Helmfile + +For a comprehensive deployment, managing all related Helm releases and their values via a single Helmfile offers simplicity and maintainability. This method is particularly effective when deploying complex stacks. + +### Deploy blockchain namespaces as desired + +:::note +If you have existing external blockchain nodes that you would like to use instead of deploying them into your cluster, you can skip this section, but make sure that you can access those nodes securely (e.g. via an internal network, or using HTTPS and authentication). +::: + +#### (optional, arbitrum-sepolia) Install Arbitrum Nitro and Proxyd for Arbitrum Sepolia + +The following `helmfile.yaml` provides an example configuration for deploying Arbitrum Nitro on the Arbitrum Sepolia network. For an easier setup process, we recommend utilizing the [Launchpad Arbitrum namespace](#optional-arbitrum-sepolia-install-arbitrum-nitro-and-proxyd-for-arbitrum-sepolia), which includes most of the necessary configurations pre-defined for your convenience. + +```yaml +# helmfile.yaml +repositories: + - name: graphops + url: https://graphops.github.io/launchpad-charts + +releases: + - name: arbitrum-nitro + namespace: arbitrum-sepolia + createNamespace: true + chart: graphops/arbitrum-nitro + version: 0.3.4 + values: + - nitro: + config: + chain: 421614 # determines Arbitrum network - 421614 Sepolia + parentChainUrl: http://your-eth-sepolia-url:8545 ## changeme + parentChainBeaconUrl: http://your-eth-consensus-node-url:5052 ## changeme + + volumeClaimSpec: + resources: + requests: + # -- The amount of disk space to provision for Arbitrum Nitro + storage: 1Ti + # -- The storage class to use when provisioning a persistent volume for Arbitrum-Nitro + storageClassName: openebs-rawfile-localpv # change me as needed + + restoreSnapshot: + enabled: false + + extraLabels: + app.kubernetes.io/workload-type: blockchain-stateful + app.kubernetes.io/blockchain: arbitrum-nitro + + # if using Prometheus for monitoring: + prometheus: + serviceMonitors: + enabled: true + + - name: proxyd-nitro + namespace: arbitrum-sepolia + createNamespace: true + chart: graphops/proxyd + version: 0.5.3 + values: + - backends: + arbitrum-nitro: + enabled: true + # -- Define the RPC URL for the backend + rpcUrl: http://arbitrum-nitro:8547 + # -- Define the WS URL for the backend + wsUrl: ws://arbitrum-nitro:8548 + # -- Define additional configuration keys for the backend (see [proxyd config](https://github.com/ethereum-optimism/optimism/blob/5d309e6a6d5e1ef6a88c1ce827b7e6d47f033bbb/proxyd/example.config.toml#L47)) + extraConfig: + consensus_skip_peer_count: true + # -- Define which backend groups the backend is part of + groups: + - main + + # if using Prometheus and Grafana for monitoring: + prometheus: + serviceMonitors: + enabled: true + + grafana: + dashboards: true +``` + + +Deploy by syncing your cluster with the declarative `helmfile.yaml`: + +```shell +helmfile -f path/to/helmfile.yaml sync +``` + +### Install the Graph Arbitrum Sepolia Indexer Stack + +This section of the guide does not include the setup for `subgraph-data` and `indexer-metadata` PostgreSQL databases necessary for `graph-node` and `indexer-agent`. You are encouraged to explore [managed solutions](https://www.postgresql.org/support/professional_hosting/), use [Bitnami's chart](https://github.com/bitnami/charts/tree/main/bitnami/postgresql), or deploy [Zalando's Operator](https://github.com/zalando/postgres-operator/tree/master) as part of the Launchpad Namespaces which includes a ready-to-use Postgres setup or independently. + +Include the necessary configurations for `graph-node` and `indexer-agent` in your helmfile.yaml as shown in the previous sections, adjusting PostgreSQL references and other settings to fit your specific requirements. + +```yaml +releases: + - name: graph-node + namespace: arbitrum-sepolia + createNamespace: true + chart: graphops/graph-node + version: 0.5.3 + values: + # This is a values.yaml override file for https://github.com/graphops/launchpad-charts/tree/main/charts/graph-node + - graphNodeDefaults: + env: + # Graph Node configuration + IPFS: "https://ipfs.network.thegraph.com" + GRAPH_ALLOW_NON_DETERMINISTIC_FULLTEXT_SEARCH: "true" + # Database configuration + PRIMARY_SUBGRAPH_DATA_PGHOST: ## change me + PRIMARY_SUBGRAPH_DATA_PGPORT: 5432 + PRIMARY_SUBGRAPH_DATA_PGDATABASE: ## change me + + # Database sensitive/secret information + secretEnv: + PRIMARY_SUBGRAPH_DATA_PGUSER: + secretName: + key: username + PRIMARY_SUBGRAPH_DATA_PGPASSWORD: + secretName: + key: password + + graphNodeGroups: + index: + replicaCount: 1 # scale me + query: + replicaCount: 1 # scale me + + chains: + mainnet: + enabled: true + shard: primary + provider: + - label: eth-mainnet + url: ## change me + features: [archive, traces] + + arbitrum-sepolia: + enabled: true + shard: primary + provider: + - label: arbitrum-sepolia + url: http://proxyd-proxyd.arbitrum-sepolia:8545 + features: [archive, traces] + + # if using Prometheus and Grafana for monitoring: + prometheus: + serviceMonitors: + enabled: true + + grafana: + dashboards: true + datasources: true + + - name: graph-network-indexer + namespace: arbitrum-sepolia + createNamespace: true + chart: graphops/graph-network-indexer + version: 0.2.5 + values: + # This is a values.yaml override file for https://github.com/graphops/launchpad-charts/tree/main/charts/graph-network-indexer + - indexerDefaults: + config: + ethereum: "http://proxyd-proxyd.arbitrum-sepolia:8545" + ethereum-network: "arbitrum-sepolia" + network-subgraph-endpoint: "https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-arbitrum-sepolia" + graph-node-query-endpoint: "http://graph-node-query:8000" + graph-node-status-endpoint: "http://graph-node-block-ingestor:8030/graphql" + postgres-host: "" ## change me + postgres-database: "" ## change me + + indexerAgent: + config: + collect-receipts-endpoint: "https://gateway-testnet-arbitrum.network.thegraph.com/collect-receipts" + network-subgraph-deployment: "QmT8UDGK7zKd2u2NQZwhLYHdA4KM55QsivkE3ouCuX6fEj" # find at https://github.com/graphprotocol/indexer/blob/main/docs/networks.md + epoch-subgraph-endpoint: "https://api.thegraph.com/subgraphs/name/graphprotocol/arbitrum-sepolia-ebo" # find at https://github.com/graphprotocol/indexer/blob/main/docs/networks.md + epoch-subgraph-deployment: "QmTpu2mVquoMpr4SWSM77nGkU3tcUS1Bhk1sVHpjDrAUAx" + graph-node-admin-endpoint: "http://graph-node-block-ingestor:8020" + public-indexer-url: "" ## change me + index-node-ids: "graph-node-index-0" # if more than one graph-node index, specify as comma delimited list ie "graph-node-index-0, graph-node-index-1" + + secretEnv: + INDEXER_AGENT_MNEMONIC: + secretName: + key: mnemonic + INDEXER_AGENT_POSTGRES_USERNAME: + secretName: + key: username + INDEXER_AGENT_POSTGRES_PASSWORD: + secretName: + key: password + + + indexerService: + replicas: 1 # scale me + + config: + client-signer-address: "0xe1EC4339019eC9628438F8755f847e3023e4ff9c" # find at https://github.com/graphprotocol/indexer/blob/main/docs/networks.md + + secretEnv: + INDEXER_SERVICE_MNEMONIC: + secretName: + key: mnemonic + INDEXER_SERVICE_POSTGRES_USERNAME: + secretName: + key: username + INDEXER_SERVICE_POSTGRES_PASSWORD: + secretName: + key: password + # if using Prometheus and Grafana for monitoring: + prometheus: + serviceMonitors: + enabled: true + + grafana: + dashboards: true + + - name: subgraph-radio + namespace: arbitrum-sepolia + createNamespace: true + chart: graphops/subgraph-radio + version: 0.2.8 + values: + - env: + GRAPH_NODE_STATUS_ENDPOINT: http://graph-node-block-ingestor:8030/graphql + INDEXER_MANAGEMENT_SERVER_ENDPOINT: http://graph-network-indexer-agent:8000 + GRAPHCAST_NETWORK: "testnet" + REGISTRY_SUBGRAPH: https://api.thegraph.com/subgraphs/name/hopeyen/graphcast-registry-arb-se + NETWORK_SUBGRAPH: https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-arbitrum-sepolia + secretEnv: + MNEMONIC: + secretName: + key: mnemonic + + - name: graph-toolbox + namespace: arbitrum-sepolia + createNamespace: true + chart: graphops/graph-toolbox + version: 0.1.0 + values: + - config: + graphNode: + # -- URL to Graph Node Admin API + adminApiUrl: http://graph-node-block-ingestor:8020 + existingConfigMap: + # -- The name of the ConfigMap that contains your Graph Node config.toml + configMapName: graph-node-config + # -- The name of the data key in the ConfigMap that contains your config.toml + configFileKey: config.toml + indexer: + # -- URL to Indexer Agent Management Server + indexerAgentManagementUrl: http://graph-network-indexer-agent:8000 + + aliases: + graphman: graphman --config /graphman-config/config.toml + indexer: graph-indexer indexer + psql-primary-subgraph-data: > + PGPASSWORD=$PRIMARY_SUBGRAPH_DATA_PGPASSWORD psql -w -U $PRIMARY_SUBGRAPH_DATA_PGUSER -d "host=$PRIMARY_SUBGRAPH_DATA_PGHOST,port=$PRIMARY_SUBGRAPH_DATA_PGPORT,dbname=$PRIMARY_SUBGRAPH_DATA_PGDATABASE" + psql-indexer-metadata: > + PGPASSWORD=$INDEXER_METDATA_PGPASSWORD psql -w -U $INDEXER_METADATA_PGUSER -d "host=$INDEXER_METADATA_PGHOST,port=$INDEXER_METADATA_PGPORT,dbname=$INDEXER_METADATA_PGDATABASE" + + env: + PRIMARY_SUBGRAPH_DATA_PGHOST: ## change me + PRIMARY_SUBGRAPH_DATA_PGPORT: 5432 + PRIMARY_SUBGRAPH_DATA_PGDATABASE: ## change me + INDEXER_METADATA_PGHOST: ## change me + INDEXER_METADATA_PGPORT: 5432 + INDEXER_METADATA_PGDATABASE: ## change me + + secretEnv: + PRIMARY_SUBGRAPH_DATA_PGUSER: + secretName: ## change me + key: username + PRIMARY_SUBGRAPH_DATA_PGPASSWORD: + secretName: ## change me + key: password + INDEXER_METADATA_PGUSER: + secretName: ## change me + key: username + INDEXER_METDATA_PGPASSWORD: + secretName: ## change me + key: password +``` + +Proceed to deploy: + +```shell +helmfile -f path/to/helmfile.yaml sync +``` + +### 🎉 Milestone: Graph Indexer running and accessible + +Once your deployments are successfully applied, your Graph Indexer should be operational, with blockchain nodes (if deployed) and the Graph Indexing stack running in your Kubernetes cluster. + +- [x] We (optionally) configured and deployed blockchain nodes into our cluster +- [x] We configured and deployed the Graph Indexing stack into our cluster +- [ ] Next: Use the remote-toolbox to allocate to subgraphs and begin serving requests diff --git a/docs/launchpad/tutorials/arbitrum-archive-kubernetes-guide.md b/docs/launchpad/tutorials/arbitrum-archive-kubernetes-guide.md index 24301b89..cabe9c9d 100644 --- a/docs/launchpad/tutorials/arbitrum-archive-kubernetes-guide.md +++ b/docs/launchpad/tutorials/arbitrum-archive-kubernetes-guide.md @@ -90,7 +90,7 @@ arbitrum: ```sh helm repo add graphops http://graphops.github.io/launchpad-charts -helm install --dry-run arbitrum-classic graphops/arbitrum-classic:latest --namespace arbitrum-one --value arbitrum-classic.yaml +helm install --dry-run arbitrum-classic graphops/arbitrum-classic:latest --namespace arbitrum-one --values arbitrum-classic.yaml ``` ### Deploy Arbitrum Nitro @@ -110,5 +110,5 @@ nitro: 2. Deploy using helm: ```sh -helm install --dry-run arbitrum-nitro graphops/arbitrum-classic:latest --namespace arbitrum-one --value arbitrum-nitro.yaml +helm install --dry-run arbitrum-nitro graphops/arbitrum-classic:latest --namespace arbitrum-one --values arbitrum-nitro.yaml ``` diff --git a/src/components/HomepageFeatures.tsx b/src/components/HomepageFeatures.tsx index 67d64666..6870a3c4 100644 --- a/src/components/HomepageFeatures.tsx +++ b/src/components/HomepageFeatures.tsx @@ -27,7 +27,7 @@ const FeatureList: FeatureItem[] = [ Launchpad provides a toolbox for smoothly operating your Graph Protocol Indexer on Kubernetes ), - link: "/launchpad/intro", + link: "/launchpad/docs-map", }, { title: "Join Graphcast to coordinate with other Indexers using Radios",