From a9c50f1f22bcc8e5aa10c8b02b737a4bd86d4d89 Mon Sep 17 00:00:00 2001 From: Sean Yang Date: Thu, 11 Jan 2024 16:16:19 -0800 Subject: [PATCH 01/39] updates examples, add version matching note (#2278) --- docs/example_applications_algorithms.rst | 27 ++++++++- docs/getting_started.rst | 3 + docs/release_notes/flare_240.rst | 11 ++-- .../communication_configuration.rst | 3 + examples/README.md | 6 +- examples/hello-world/step-by-step/README.md | 56 ++++--------------- .../step-by-step/cifar10/README.md | 22 +++++--- .../sag_deploy_map.ipynb | 0 .../hello-world/step-by-step/higgs/README.md | 23 ++++---- 9 files changed, 77 insertions(+), 74 deletions(-) rename examples/hello-world/step-by-step/cifar10/{sag_with_deploy_map => sag_deploy_map}/sag_deploy_map.ipynb (100%) diff --git a/docs/example_applications_algorithms.rst b/docs/example_applications_algorithms.rst index 7f4b5c17e4..d93b07ce79 100644 --- a/docs/example_applications_algorithms.rst +++ b/docs/example_applications_algorithms.rst @@ -9,8 +9,31 @@ NVIDIA FLARE has several tutorials and examples to help you get started with fed 1. Step-By-Step Example Series ============================== - * :github_nvflare_link:`Step-by-Step CIFAR-10 Examples (GitHub) ` - Step-by-step examples series with CIFAR-10 (image data) to showcase to showcase different FLARE features, workflows, and APIs. - * :github_nvflare_link:`Step-by-Step HIGGS Examples (GitHub) ` - Step-by-step examples series with HIGGS (tabular data) to showcase to showcase different FLARE features, workflows, and APIs. +:github_nvflare_link:`Step-by-Step Examples (GitHub) ` - Step-by-step examples series with CIFAR-10 (image data) and HIGGS (tabular data) to showcase different FLARE features, workflows, and APIs. + +1.1 CIFAR-10 Image Data Examples +-------------------------------- + + * :github_nvflare_link:`image_stats ` - federated statistics (histograms) of CIFAR10. + * :github_nvflare_link:`sag ` - scatter and gather (SAG) workflow with PyTorch with Client API. + * :github_nvflare_link:`sag_deploy_map ` - scatter and gather workflow with deploy_map configuration for deployment of apps to different sites using the Client API. + * :github_nvflare_link:`sag_model_learner ` - scatter and gather workflow illustrating how to write client code using the ModelLearner. + * :github_nvflare_link:`sag_executor ` - scatter and gather workflow demonstrating show to write client-side executors. + * :github_nvflare_link:`sag_mlflow ` - MLflow experiment tracking logs with the Client API in scatter & gather workflows. + * :github_nvflare_link:`sag_he ` - homomorphic encyption using Client API and POC -he mode. + * :github_nvflare_link:`cse ` - cross-site evaluation using the Client API. + * :github_nvflare_link:`cyclic ` - cyclic weight transfer workflow with server-side controller. + * :github_nvflare_link:`cyclic_ccwf ` - client-controlled cyclic weight transfer workflow with client-side controller. + * :github_nvflare_link:`swarm ` - swarm learning and client-side cross-site evaluation with Client API. + +1.2 HIGGS Tabular Data Examples +------------------------------- + + * :github_nvflare_link:`tabular_stats `- federated stats tabular histogram calculation. + * :github_nvflare_link:`sklearn_linear `- federated linear model (logistic regression on binary classification) learning on tabular data. + * :github_nvflare_link:`sklearn_svm `- federated SVM model learning on tabular data. + * :github_nvflare_link:`sklearn_kmeans `- federated k-Means clustering on tabular data. + * :github_nvflare_link:`xgboost `- federated horizontal xgboost learning on tabular data with bagging collaboration. 2. Hello World Examples ======================= diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 1abb87c58d..d5bb9cb8ab 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -63,6 +63,9 @@ establishing a secure, distributed FL workflow. Installation ============= +.. note:: + The server and client versions of nvflare must match, we do not support cross-version compatibility. + Python Version -------------- diff --git a/docs/release_notes/flare_240.rst b/docs/release_notes/flare_240.rst index 80eb65d067..97d03179f6 100644 --- a/docs/release_notes/flare_240.rst +++ b/docs/release_notes/flare_240.rst @@ -83,20 +83,21 @@ Each example will build upon previous ones to showcase different features, workf **CIFAR10 Examples:** -- stats: federated statistics (histograms) of CIFAR10. +- image_stats: federated statistics (histograms) of CIFAR10. - sag: scatter and gather (SAG) workflow with PyTorch with Client API. -- sag_with_deploy_map: scatter and gather workflow with deploy_map configuration, for deployment of apps to different sites using the Client API. -- cse: cross-site evaluation using the Client API. +- sag_deploy_map: scatter and gather workflow with deploy_map configuration, for deployment of apps to different sites using the Client API. - sag_model_learner: scatter and gather workflow illustrating how to write client code using the ModelLearner. - sag_executor: scatter and gather workflow demonstrating show to write client-side executors. +- sag_mlflow: MLflow experiment tracking logs with the Client API in scatter & gather workflows. +- sag_he: homomorphic encyption using Client API and POC -he mode. +- cse: cross-site evaluation using the Client API. - cyclic: cyclic weight transfer workflow with server-side controller. - cyclic_ccwf: client-controlled cyclic weight transfer workflow with client-side controller. - swarm: swarm learning and client-side cross-site evaluation with Client API. -- sag_with_mlflow: MLflow experiment tracking logs with the Client API in scatter & gather workflows. **HIGGS Examples:** -- tabular_stats: federated stats tabular histogram calculation. +- tabular_stats: federated statistics tabular histogram calculation. - scikit_learn: federated linear model (logistic regression on binary classification) learning on tabular data. - sklearn_svm: federated SVM model learning on tabular data. - sklearn_kmeans: federated k-Means clustering on tabular data. diff --git a/docs/user_guide/configurations/communication_configuration.rst b/docs/user_guide/configurations/communication_configuration.rst index 4cdd95f0ce..dbe1ad2a4b 100644 --- a/docs/user_guide/configurations/communication_configuration.rst +++ b/docs/user_guide/configurations/communication_configuration.rst @@ -126,6 +126,9 @@ This is done by setting use_aio_grpc to true: ``"use_aio_grpc": true`` +On the server side if you use the non-AIO gRPC driver, the default maximum number of workers is 100, meaning that there can be at most 100 concurrent connections to the server. +If this is not enough, you will need to use the AIO gRPC driver. + Ad-hoc Connections ================== diff --git a/examples/README.md b/examples/README.md index 7887865d53..a8b4b577a1 100644 --- a/examples/README.md +++ b/examples/README.md @@ -77,9 +77,11 @@ When you open a notebook, select the kernel `nvflare_example` using the dropdown |---------|---------|-----------------|-----------------|-----------|---------| | [image_stats](./hello-world/step-by-step/cifar10/stats/image_stats.ipynb) | CIFAR10 | server | Executor | Pandas | Example for federated stats image histogram calculation. | | [sag](./hello-world/step-by-step/cifar10/sag/sag.ipynb) | CIFAR10 | server | Client API| PyTorch | Example for FedAvg with [ScatterAndGather](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.scatter_and_gather.html) controller workflow using the Client API. | -| [sag_with_deploy_map](./hello-world/step-by-step/cifar10/sag_with_deploy_map/sag_deploy_map.ipynb) | CIFAR10 | server | Client API | PyTorch | Example showcasing site-specific configurations and deploy_map. | -| [sag_executor](./hello-world/step-by-step/cifar10/sag_executor/sag_executor.ipynb) | CIFAR10 | server | Executor | PyTorch | Example with [ScatterAndGather](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.scatter_and_gather.html) using an Executor. | +| [sag_deploy_map](./hello-world/step-by-step/cifar10/sag_deploy_map/sag_deploy_map.ipynb) | CIFAR10 | server | Client API | PyTorch | Example showcasing site-specific configurations and deploy_map. | | [sag_model_learner](./hello-world/step-by-step/cifar10/sag_model_learner/sag_model_learner.ipynb) | CIFAR10 | server | ModelLearner | PyTorch | Example with [ScatterAndGather](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.scatter_and_gather.html) using a ModelLearner. | +| [sag_executor](./hello-world/step-by-step/cifar10/sag_executor/sag_executor.ipynb) | CIFAR10 | server | Executor | PyTorch | Example with [ScatterAndGather](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.scatter_and_gather.html) using an Executor. | +| [sag_mlflow](./hello-world/step-by-step/cifar10/sag_mlflow/sag_mlflow.ipynb) | CIFAR10 | server | Client API | PyTorch | MLflow experiment tracking logs with [ScatterAndGather](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.scatter_and_gather.html) using the Client API. | +| [sag_he](./hello-world/step-by-step/cifar10/sag_he/sag_he.ipynb) | CIFAR10 | server | Client API | PyTorch | Example with homomorphic encyption using Client API and POC -he mode. | | [cse](./hello-world/step-by-step/cifar10/cse/cse.ipynb) | CIFAR10 | server | Client API| PyTorch | Example using [CrossSiteModelEval](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.cross_site_model_eval.html) controller workflow. | | [cyclic](./hello-world/step-by-step/cifar10/cyclic/cyclic.ipynb) | CIFAR10 | server | Client API | PyTorch | Example for cyclic weight transfer using [CyclicController](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.cyclic_ctl.html) controller workflow. | | [cyclic_ccwf](./hello-world/step-by-step/cifar10/cyclic_ccwf/cyclic_ccwf.ipynb) | CIFAR10 | client| Client API | PyTorch | Example for client-controlled cyclic weight transfer using [CyclicClientController](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.ccwf.cyclic_client_ctl.html) controller workflow. | diff --git a/examples/hello-world/step-by-step/README.md b/examples/hello-world/step-by-step/README.md index 4e563122b2..77ecc34f52 100644 --- a/examples/hello-world/step-by-step/README.md +++ b/examples/hello-world/step-by-step/README.md @@ -1,6 +1,10 @@ # Step-by-Step Examples -When given a machine learning problem, we probably wonder, where do we start to formulate the federated learning problem. +These step-by-step example series are aimed to help users quickly get started and learn about FLARE. +For consistency, each example in the series uses the same dataset- CIFAR10 for image data and the HIGGS dataset for tabular data. +The examples will build upon previous ones to showcase different features, workflows, or APIs, allowing users to gain a comprehensive understanding of FLARE functionalities. + +Given a machine learning problem, here are some common questions we aim to cover when formulating a federated learning problem: * What does the data look like? * How do we compare global statistics with the site's local data statistics? @@ -9,52 +13,12 @@ When given a machine learning problem, we probably wonder, where do we start to * Given the formulation, how to convert the existing machine learning or deep learning code to Federated learning code. * [ML to FL examples](https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/ml-to-fl/README.md) * For different types of federated learning workflows: Scatter and Gather, Cyclic Weight Transfer, Swarming learning, -Vertical learning, ..., what do we need to change ? -* Further how can apply the experiment log, so all sites' metrics and global metrics can be viewed -* in experiment tracking tools such as Weights & Biases, MLFLow, or simply Tensorboard - -In this "step-by-step" examples, we will dive these questions in two series of examples: - -## Multi-class classification with image data using CIFAR10 dataset - -The CIFAR10 dataset has the following 10 classes: ‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’, ‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’. -The images in CIFAR-10 are of size 3x32x32, i.e. 3-channel color images of 32x32 pixels in size. - -![image](cifar10/data/cifar10.png) - -We will use the [pytorch](https://pytorch.org/) deep learning framework to illustrate how to formulate and convert the deep learning training -program to a federated learning training program. The example will include: - -* Federated Histogram analysis with Federated Statistics -* Scatter and Gather (SAG) workflow with NVFLARE Client APIs -* Cyclic Weight Transfer workflow with NVFLARE Client APIs -* Swarm Learning Workflow with NVFLARE Client APIs -* SAG with NVFLARE model learner APIs -* SAG with NVFLARE Executor APIs -* SAG with NVFLARE Client APIs + MLflow - - -## Binary classification with tabular data using HIGGS dataset - -### HIGGS Dataset - -[HIGGS dataset](https://archive.ics.uci.edu/dataset/280/higgs) contains 11 million instances, each with 28 attributes, for binary classification to predict whether an event corresponds to the decayment of a Higgs boson or not. - -The first 21 features (columns 2-22) are kinematic properties measured by the particle detectors in the accelerator. -The data has been produced using Monte Carlo simulations. The first 21 features are kinematic properties measured by the particle detectors in the accelerator. The last 7 features are functions of the first 21 features; these are high-level features derived by physicists to help discriminate between the two classes. +Vertical learning, ... what do we need to change ? +* How can we capture the experiment log, so all sites' metrics and global metrics can be viewed in experiment tracking tools such as Weights & Biases, MLfLow, or Tensorboard -Please note that the [UCI's website](https://archive.ics.uci.edu/dataset/280/higgs) may experience occasional downtime. +In these "step-by-step" examples, we will dive into these questions in two series of examples (See the README in each directory for more details about each series): -With the HIGGs Dataset, we like to demonstrate traditional machine learning techniques in federated learning. -These include: +* [cifar10](cifar10) - Multi-class classification with image data using CIFAR10 dataset +* [higgs](higgs) - Binary classification with tabular data using HIGGS dataset -* Federated Statistics for tabular data -* Federated Logistic Regression -* Federated Kmeans -* Federated SVM -* Federated Horizontal XGBoost -These examples demostrate: -* How to use the NVFlare Client APIs to convert the traditional machine learning code to federated learning code. Most of them contains local training scripts as baselines for comparison. -* How different machine learning methods can be applied to the same problem. Different behaviors and accuracies can be observed, as a reference for choosing the right method for the problem. -* How federated learning impacts different machine learning methods. Some methods are more sensitive to the federated learning process, and some are less. diff --git a/examples/hello-world/step-by-step/cifar10/README.md b/examples/hello-world/step-by-step/cifar10/README.md index 245542096c..d90d7471ec 100644 --- a/examples/hello-world/step-by-step/cifar10/README.md +++ b/examples/hello-world/step-by-step/cifar10/README.md @@ -1,5 +1,5 @@ -# Training a image classifier with CIFAR10 data +# Training an image classifier with CIFAR10 dataset We will use the original [Training a Classifer](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html) example in pytorch as the code base. @@ -9,12 +9,16 @@ The images in CIFAR-10 are of size 3x32x32, i.e. 3-channel color images of 32x32 ![image](data/cifar10.png) -In the follow examples, we will show various Federated Learning workflows. +In the following examples, we will show various Federated Learning workflows and features: -* [Image intensity histogram caculation](stats) -* Scatter and Gather (SAG) workflow with NVFLARE Client APIs -* Cyclic Weight Transfer workflow with NVFLARE Client APIs -* Swarm Learning Workflow with NVFLARE Client APIs -* SAG with NVFLARE model learner APIs -* SAG with NVFLARE Executor APIs -* SAG with NVFLARE Client APIs + MLflow +* [stats](stats) - Federated statistics image intensity histogram calculation. +* [sag](sag) - Scatter and Gather (SAG) workflow with Client API +* [sag_deploy_map](sag_deploy_map) - SAG with deploy_map configuration for deployment of apps to different sites. +* [sag_model_learner](sag_model_learner) - SAG with Model Learner API +* [sag_executor](sag_executor) - SAG with Executor API +* [sag_mlflow](sag_mlflow) - SAG with MLflow experiment tracking logs. +* [sag_he](sag_he) - SAG with homomorphic encyption using POC -he mode. +* [cse](cse) - Cross-site evaluation with server-side controller. +* [cyclic](cyclic) - Cyclic Weight Transfer (cyclic) workflow with server-side controller. +* [cyclic_ccwf](cyclic_ccwf) - Client-controlled cyclic workflow with client-side controller. +* [swarm](swarm) - Swarm learning and client-controlled cross-site evaluation. diff --git a/examples/hello-world/step-by-step/cifar10/sag_with_deploy_map/sag_deploy_map.ipynb b/examples/hello-world/step-by-step/cifar10/sag_deploy_map/sag_deploy_map.ipynb similarity index 100% rename from examples/hello-world/step-by-step/cifar10/sag_with_deploy_map/sag_deploy_map.ipynb rename to examples/hello-world/step-by-step/cifar10/sag_deploy_map/sag_deploy_map.ipynb diff --git a/examples/hello-world/step-by-step/higgs/README.md b/examples/hello-world/step-by-step/higgs/README.md index 09baafba9b..877a0fb118 100644 --- a/examples/hello-world/step-by-step/higgs/README.md +++ b/examples/hello-world/step-by-step/higgs/README.md @@ -1,19 +1,22 @@ -# Training traditional ML classifiers with HIGGS data +# Training traditional ML classifiers with HIGGS dataset -[HIGGS dataset](https://archive.ics.uci.edu/dataset/280/higgs) contains 11 million instances, each with 28 attributes, for binary classification to predict whether an event corresponds to the decayment of a Higgs boson or not. +The [HIGGS dataset](https://archive.ics.uci.edu/dataset/280/higgs) contains 11 million instances, each with 28 attributes, for binary classification to predict whether an event corresponds to the decayment of a Higgs boson or not. (Please note that the [UCI's website](https://archive.ics.uci.edu/dataset/280/higgs) may experience occasional downtime) The first 21 features (columns 2-22) are kinematic properties measured by the particle detectors in the accelerator. The data has been produced using Monte Carlo simulations. The first 21 features are kinematic properties measured by the particle detectors in the accelerator. The last 7 features are functions of the first 21 features; these are high-level features derived by physicists to help discriminate between the two classes. -Please note that the [UCI's website](https://archive.ics.uci.edu/dataset/280/higgs) may experience occasional downtime. +Key Concepts: +* How to use the NVFlare Client APIs to convert the traditional machine learning code to federated learning code. Most of them contains local training scripts as baselines for comparison. +* How different machine learning methods can be applied to the same problem. Different behaviors and accuracies can be observed, as a reference for choosing the right method for the problem. +* How federated learning impacts different machine learning methods. Some methods are more sensitive to the federated learning process, and some are less. -With the HIGGs Dataset, in the following examples, we like to demonstrate traditional machine learning techniques in federated learning. -These include: +In the following examples, we will demonstrate traditional machine learning techniques with tabular data for federated learning: + +* [stats](stats) - Federated statistics for tabular histogram calculation. +* [sklearn-linear](sklearn-linear) - Federated linear model (logistic regression on binary classification) learning. +* [sklearn-svm](sklearn-svm) - Federated SVM model learning. +* [sklearn-kmeans](sklearn-kmeans) - Federated k-Means clustering. +* [xgboost](xgboost) - Federated horizontal xgboost learning with bagging collaboration. -* Federated Statistics for tabular data -* Federated Logistic Regression -* Federated Kmeans -* Federated SVM -* Federated Horizontal XGBoost From 6bd1ed537d287fe332cd2b32500b8006d4910740 Mon Sep 17 00:00:00 2001 From: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Date: Thu, 11 Jan 2024 19:55:10 -0800 Subject: [PATCH 02/39] increase max_login_tries from 5 to 15 and increase the sleep interval from 1 to 1.5 sec (#2279) --- nvflare/fuel/hci/client/api.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nvflare/fuel/hci/client/api.py b/nvflare/fuel/hci/client/api.py index 0597ee7a2b..492b11a2f0 100644 --- a/nvflare/fuel/hci/client/api.py +++ b/nvflare/fuel/hci/client/api.py @@ -46,7 +46,7 @@ _CMD_TYPE_SERVER = 2 MAX_AUTO_LOGIN_TRIES = 300 -AUTO_LOGIN_INTERVAL = 1.0 +AUTO_LOGIN_INTERVAL = 1.5 class ResultKey(object): @@ -313,7 +313,7 @@ def __init__( session_timeout_interval=None, session_status_check_interval=None, auto_login_delay: int = 5, - auto_login_max_tries: int = 5, + auto_login_max_tries: int = 15, event_handlers=None, ): """API to keep certs, keys and connection information and to execute admin commands through do_command. From c806e1e10a9a19402d6446607a9bb4c154376015 Mon Sep 17 00:00:00 2001 From: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Date: Thu, 11 Jan 2024 19:55:47 -0800 Subject: [PATCH 03/39] handle none value for the subcommand parser (#2276) --- nvflare/cli.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/nvflare/cli.py b/nvflare/cli.py index 4319ed9608..316973f8ba 100644 --- a/nvflare/cli.py +++ b/nvflare/cli.py @@ -143,7 +143,8 @@ def parse_args(prog_name: str): if argv: msg = f"{prog_name} {cmd}: unrecognized arguments: {' '.join(argv)}\n" print(f"\nerror: {msg}") - sub_cmd_parser.print_help() + if sub_cmd_parser: + sub_cmd_parser.print_help() _parser.exit(2, "\n") return _parser, _parser.parse_args(), sub_cmd_parsers From 304c1d866cf4a1093ab3c2fe3398e0e62791c450 Mon Sep 17 00:00:00 2001 From: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Date: Tue, 16 Jan 2024 09:33:27 -0800 Subject: [PATCH 04/39] Address VDR comments [slip ci] (#2283) * Add VDR suggestions * fix typo --- docs/example_applications_algorithms.rst | 68 +++++++++--------- docs/user_guide/nvflare_cli/poc_command.rst | 4 +- examples/README.md | 22 +++--- .../cifar10/sag/fed_avg_one_round.png | Bin 0 -> 66180 bytes .../step-by-step/cifar10/sag/mpi_gather.png | Bin 0 -> 7653 bytes .../step-by-step/cifar10/sag/mpi_scatter.png | Bin 0 -> 16160 bytes .../step-by-step/cifar10/sag/sag.ipynb | 41 +++++++---- .../sag_model_learner/sag_model_learner.ipynb | 8 +-- examples/tutorials/setup_poc.ipynb | 17 ++--- 9 files changed, 86 insertions(+), 74 deletions(-) create mode 100644 examples/hello-world/step-by-step/cifar10/sag/fed_avg_one_round.png create mode 100644 examples/hello-world/step-by-step/cifar10/sag/mpi_gather.png create mode 100644 examples/hello-world/step-by-step/cifar10/sag/mpi_scatter.png diff --git a/docs/example_applications_algorithms.rst b/docs/example_applications_algorithms.rst index d93b07ce79..820e9dc7f9 100644 --- a/docs/example_applications_algorithms.rst +++ b/docs/example_applications_algorithms.rst @@ -6,36 +6,7 @@ Example Applications NVIDIA FLARE has several tutorials and examples to help you get started with federated learning and to explore certain features in the :github_nvflare_link:`examples directory `. -1. Step-By-Step Example Series -============================== - -:github_nvflare_link:`Step-by-Step Examples (GitHub) ` - Step-by-step examples series with CIFAR-10 (image data) and HIGGS (tabular data) to showcase different FLARE features, workflows, and APIs. - -1.1 CIFAR-10 Image Data Examples --------------------------------- - - * :github_nvflare_link:`image_stats ` - federated statistics (histograms) of CIFAR10. - * :github_nvflare_link:`sag ` - scatter and gather (SAG) workflow with PyTorch with Client API. - * :github_nvflare_link:`sag_deploy_map ` - scatter and gather workflow with deploy_map configuration for deployment of apps to different sites using the Client API. - * :github_nvflare_link:`sag_model_learner ` - scatter and gather workflow illustrating how to write client code using the ModelLearner. - * :github_nvflare_link:`sag_executor ` - scatter and gather workflow demonstrating show to write client-side executors. - * :github_nvflare_link:`sag_mlflow ` - MLflow experiment tracking logs with the Client API in scatter & gather workflows. - * :github_nvflare_link:`sag_he ` - homomorphic encyption using Client API and POC -he mode. - * :github_nvflare_link:`cse ` - cross-site evaluation using the Client API. - * :github_nvflare_link:`cyclic ` - cyclic weight transfer workflow with server-side controller. - * :github_nvflare_link:`cyclic_ccwf ` - client-controlled cyclic weight transfer workflow with client-side controller. - * :github_nvflare_link:`swarm ` - swarm learning and client-side cross-site evaluation with Client API. - -1.2 HIGGS Tabular Data Examples -------------------------------- - - * :github_nvflare_link:`tabular_stats `- federated stats tabular histogram calculation. - * :github_nvflare_link:`sklearn_linear `- federated linear model (logistic regression on binary classification) learning on tabular data. - * :github_nvflare_link:`sklearn_svm `- federated SVM model learning on tabular data. - * :github_nvflare_link:`sklearn_kmeans `- federated k-Means clustering on tabular data. - * :github_nvflare_link:`xgboost `- federated horizontal xgboost learning on tabular data with bagging collaboration. - -2. Hello World Examples +1. Hello World Examples ======================= Can be run from the :github_nvflare_link:`hello_world notebook `. @@ -45,12 +16,12 @@ Can be run from the :github_nvflare_link:`hello_world notebook ` - Example for converting Deep Learning (DL) to Federated Learning (FL) using the Client API. -2.2. Workflows +1.2. Workflows -------------- * :ref:`Hello Scatter and Gather ` - Example using the Scatter And Gather (SAG) workflow with a Numpy trainer @@ -59,13 +30,44 @@ Can be run from the :github_nvflare_link:`hello_world notebook ` - Example using Swarm Learning and Client-Controlled Cross-site Evaluation workflows. * :github_nvflare_link:`Client-Controlled Cyclic Weight Transfer ` - Example using Client-Controlled Cyclic workflow using Client API. -2.3. Deep Learning +1.3. Deep Learning ------------------ * :ref:`Hello PyTorch ` - Example image classifier using FedAvg and PyTorch as the deep learning training framework * :ref:`Hello TensorFlow ` - Example image classifier using FedAvg and TensorFlow as the deep learning training frameworks + +2. Step-By-Step Example Series +============================== + +:github_nvflare_link:`Step-by-Step Examples (GitHub) ` - Step-by-step examples series with CIFAR-10 (image data) and HIGGS (tabular data) to showcase different FLARE features, workflows, and APIs. + +2.1 CIFAR-10 Image Data Examples +-------------------------------- + + * :github_nvflare_link:`image_stats ` - federated statistics (histograms) of CIFAR10. + * :github_nvflare_link:`sag ` - scatter and gather (SAG) workflow with PyTorch with Client API. + * :github_nvflare_link:`sag_deploy_map ` - scatter and gather workflow with deploy_map configuration for deployment of apps to different sites using the Client API. + * :github_nvflare_link:`sag_model_learner ` - scatter and gather workflow illustrating how to write client code using the ModelLearner. + * :github_nvflare_link:`sag_executor ` - scatter and gather workflow demonstrating show to write client-side executors. + * :github_nvflare_link:`sag_mlflow ` - MLflow experiment tracking logs with the Client API in scatter & gather workflows. + * :github_nvflare_link:`sag_he ` - homomorphic encyption using Client API and POC -he mode. + * :github_nvflare_link:`cse ` - cross-site evaluation using the Client API. + * :github_nvflare_link:`cyclic ` - cyclic weight transfer workflow with server-side controller. + * :github_nvflare_link:`cyclic_ccwf ` - client-controlled cyclic weight transfer workflow with client-side controller. + * :github_nvflare_link:`swarm ` - swarm learning and client-side cross-site evaluation with Client API. + +2.2 HIGGS Tabular Data Examples +------------------------------- + + * :github_nvflare_link:`tabular_stats `- federated stats tabular histogram calculation. + * :github_nvflare_link:`sklearn_linear `- federated linear model (logistic regression on binary classification) learning on tabular data. + * :github_nvflare_link:`sklearn_svm `- federated SVM model learning on tabular data. + * :github_nvflare_link:`sklearn_kmeans `- federated k-Means clustering on tabular data. + * :github_nvflare_link:`xgboost `- federated horizontal xgboost learning on tabular data with bagging collaboration. + + 3. Tutorial Notebooks ===================== diff --git a/docs/user_guide/nvflare_cli/poc_command.rst b/docs/user_guide/nvflare_cli/poc_command.rst index 9bc0d578bf..70927bc93a 100644 --- a/docs/user_guide/nvflare_cli/poc_command.rst +++ b/docs/user_guide/nvflare_cli/poc_command.rst @@ -1,11 +1,9 @@ .. _poc_command: ***************************************** -Command for Proof Of Concept (POC) Mode +Proof Of Concept (POC) Command ***************************************** -Introduction to the POC Command -=============================== The POC command allows users to try out the features of NVFlare in a proof of concept deployment on a single machine. diff --git a/examples/README.md b/examples/README.md index a8b4b577a1..38858563a5 100644 --- a/examples/README.md +++ b/examples/README.md @@ -72,7 +72,17 @@ Start a Jupyter Lab: When you open a notebook, select the kernel `nvflare_example` using the dropdown menu at the top right. ![Selecting a JupyterLab kernel](./jupyterlab_kernel.png) -## 1. Step-by-Step Examples +## 1. Hello World Examples +| Example | Framework | Summary | +|----------------------------------------------------------------------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Notebook for Hello Examples](./hello-world/hello_world.ipynb) | - | Notebook for examples below. | +| [Hello Scatter and Gather](./hello-world/hello-numpy-sag/README.md) | Numpy | Example using [ScatterAndGather](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.scatter_and_gather.html) controller workflow. | +| [Hello Cross-Site Validation](./hello-world/hello-numpy-cross-val/README.md) | Numpy | Example using [CrossSiteModelEval](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.cross_site_model_eval.html) controller workflow. | +| [Hello Cyclic Weight Transfer](./hello-world/hello-cyclic/README.md) | PyTorch | Example using [CyclicController](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.cyclic_ctl.html) controller workflow to implement [Cyclic Weight Transfer](https://pubmed.ncbi.nlm.nih.gov/29617797/). | +| [Hello PyTorch](./hello-world/hello-pt/README.md) | PyTorch | Example using an image classifier using [FedAvg](https://arxiv.org/abs/1602.05629) and [PyTorch](https://pytorch.org/) as the deep learning training framework. | +| [Hello TensorFlow](./hello-world/hello-tf2/README.md) | TensorFlow2 | Example of using an image classifier using [FedAvg](https://arxiv.org/abs/1602.05629) and [TensorFlow](https://tensorflow.org/) as the deep learning training framework. | + +## 2. Step-by-Step Examples | Example | Dataset | Controller-Type | Client Category | Framework | Summary | |---------|---------|-----------------|-----------------|-----------|---------| | [image_stats](./hello-world/step-by-step/cifar10/stats/image_stats.ipynb) | CIFAR10 | server | Executor | Pandas | Example for federated stats image histogram calculation. | @@ -92,16 +102,6 @@ When you open a notebook, select the kernel `nvflare_example` using the dropdown | [sklearn_kmeans](./hello-world/step-by-step/higgs/sklearn-kmeans/sklearn_kmeans.ipynb) | HIGGS | server | Client API |sklearn | Example for federated k-Means clustering on tabular data. | | [xgboost](./hello-world/step-by-step/higgs/xgboost/xgboost_horizontal.ipynb) | HIGGS | server | Client API |XGBoost | Example for federated horizontal xgboost learning on tabular data with bagging collaboration. | -## 2. Hello World Examples -| Example | Framework | Summary | -|----------------------------------------------------------------------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Notebook for Hello Examples](./hello-world/hello_world.ipynb) | - | Notebook for examples below. | -| [Hello Scatter and Gather](./hello-world/hello-numpy-sag/README.md) | Numpy | Example using [ScatterAndGather](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.scatter_and_gather.html) controller workflow. | -| [Hello Cross-Site Validation](./hello-world/hello-numpy-cross-val/README.md) | Numpy | Example using [CrossSiteModelEval](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.cross_site_model_eval.html) controller workflow. | -| [Hello Cyclic Weight Transfer](./hello-world/hello-cyclic/README.md) | PyTorch | Example using [CyclicController](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.cyclic_ctl.html) controller workflow to implement [Cyclic Weight Transfer](https://pubmed.ncbi.nlm.nih.gov/29617797/). | -| [Hello PyTorch](./hello-world/hello-pt/README.md) | PyTorch | Example using an image classifier using [FedAvg](https://arxiv.org/abs/1602.05629) and [PyTorch](https://pytorch.org/) as the deep learning training framework. | -| [Hello TensorFlow](./hello-world/hello-tf2/README.md) | TensorFlow2 | Example of using an image classifier using [FedAvg](https://arxiv.org/abs/1602.05629) and [TensorFlow](https://tensorflow.org/) as the deep learning training framework. | - ## 3. Tutorial notebooks | Example | Summary | |----------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| diff --git a/examples/hello-world/step-by-step/cifar10/sag/fed_avg_one_round.png b/examples/hello-world/step-by-step/cifar10/sag/fed_avg_one_round.png new file mode 100644 index 0000000000000000000000000000000000000000..fea79a329bf6a99705c311f2395a10140a2d0415 GIT binary patch literal 66180 zcmeFZXIxX;);@}xy~V8v7K%vQh=NE{5kikjQxPK4yMS~;q=ZhuV*x=xMT)d2NDUBa zK@t*Bno@$15+EcfNDT=>fIuK{7oM}#{y+SGCtm_d=2~-(ImR>EGuM+_<|c;^ zo;=9I!*lrhHN)FHJb&!)@a(^Q;5Xot(z^C&9-erf>xNgXB3u@!Eq;r36wsWi&w{3u zglO}9q=!$~n69ANz#X~Vy9NWt`&xhJO_dK>|AW`-o6bJIbgiH}BfLb(n?HEVnqTB> z4W+Dg`u`-oZG+A$afKx-*=;G1lCwi@OO~S*&RaI4|V@#4Wx7*qbYO1@s zx`Opp`bQ7Sw4OP_oqDJ><{y_qVah;3@}wS5U5 zr_nN5L6L<4^+uDHhyF8C1~s&>wzjssy!?jbG7sG~TQ1F>ot`#2_B-;l!&uLBoYO8# zdzbsxCj8P;%kElubz%Nwn+uVhfyd+Z5Cmy3qIYTQoiajR^`XVjZrPD9r~}!G0H}an z>+K2(3JT4o%?Lhxh^=-1pqfSb#+bIw@-f17_625Ziov1lhmIk z4R$Y`^k+22aktWUQWOL{Vh(50cP7Wa%>H=vj&1h2&5n5R{2@Rth>~V0DP{b`?uJ@= ze%V|&okKMxbS!yvpFX=24ZPywRC)eod6b*i&0>W};J3T|S|-eQSg>7A+ z=s-Rpb_>MpblF8((G%pdiM1(*lZkRb!e$Xv1 zwlIR6aKTnozGO2c=l8#WtBn`N21iFb4$f2j!oBpC4Fsl-2j7~N`19;tlQ%d2wQ$A> zq!&NgaG_YnHdLm!lq?K0VYgh6Y!QhsKg#8~X9FnJ z#4ZPC4y{!VrSh|$r`bnFlVfY6cjO#2ArJ_g&2Hh+Cz9pl&z)PYDf2nd!LLI5=w=8m zy=@2{MW2S5DPwgcWivHe(71I)?!!mtiK<4cA<@wk#_umhrMk2B#RLJHMU+1&|7t4~ zMJ1cvKHk8<;c!`U%VMLB^2pz%mG4jNtSv>o>Y2%`DVt7nI{D=cM=L{{^8{T9sMbrC z#pgEo*wqt!EzVw@_kyJ6tIN2an>XUoH&=OEU;Wt7r$)s%=I-v0CxGcDIM)(+=GNCW z>I#@IveYDZeDgfe>-xL z-`Yt_)61F1a!>sRjH<9qbLoslKs5*SAH_B}2=j(v6vsxFOC_&sLn#$@#DRf!H7+MJb?1z?K~%@uWByA5TU zD|~#9JTwrcv7)#8U5O++b&&Fm$6^vb)m5sw$GIlyEYc__$|nFvUI`#UVqflG58v*! z$aK%58=P#3jEuBjW30%(3RpePCnI14`_QxWD*YvX*l+i0b^_h%(Yq7hv?dbow=mU5 zSY5X_k*VO&Dbr}Tq2{_x=NzdC>4Gn~hYxIjkC6eyt?wMRsh~Y&*tYWhxc1Z_HM#b~ z5wqm`aHB)=CuKH&Zdmn7gS%dM6i_G>KoX0N^*#mVF+;j*_?4B~dD`XfR@@H%zP)nS{?e9e{?=1Gs*RnVNAN(I4dC<6QZPA;n*0iR$p z_dNwurI#zQVG}q$jAQ0x8BN`@{J=c%IJYv_>3#G|rhX>ss`G)VX{wm}eK2CX3|!G6 zgV!nJk?`^(L823xv_mck8PPvv5y*t7i8M#XxBm!@Ftt(LL{aQ87J*szf@_~u=IECK zdjx0cYrDu0RJu2{$5?~TEaD6)RUu5C*0*8S{AQWVNiuD&u(Qw;{5b?@-i8z1>m@P( z+i1BBI+talA&+@d0S$Lke&Km0sYd)X(YTg=?ki2d{v9h~l zU?XiamBuSP#3{>h1u5r!7j7_XDEl&zk2)&u?<;6g&0UW@Js7ZQC@xD0W`rU9rQEqe z7o_jzp3h82OV*L?GbhhXIlg@*DkKgbPh1dI{ruR!F^@mMV7tN2x6yNJb@ioAK^%j+ zjQK>>XfzJX)VonV$>GXG;d@|deiosuGmJbF47Tx!yY%5frUllIAkn=^>$q}Oz_-!g z7?bI7zb)ZXX~g%)7X#Ogz+#8_r3=^3P9G&8cHc{8f{kUKf~GmIuV%+VXLU=CczvF) zUgdBD(ZkRo-+FG6xQK}yrupnEkEqdT zsY~g00X5aU1gZN^j&pf_Hq~Aju7Y^~acPWqx>9~+jD29UaL74v=GJg_$V|ZfyE?NC zkui2VwYXNOB~b*|Y@2>D?RE(de84B7dl+WLVLMC_+`k)9<}1W16n0vfCB_Y%_|P_{QIp%W?)1s}_-+3Bge5R-c0}Z30GYefg`5i4?fJoC zu{OdtsZtt+8Gm>G+4SjJG9;W* zJ0uA$;Fu?kSKObIXJZ3VgM5*cH$-x$czvBx{&|i*SyH1{kRc!u7gB$TUtZ&*^Sl%5kRJBd1)R=Etm;^n z$x*%6rx!g>ksv}(Zn`Bs__Fnco)0C3uF}n!?!JB2khRZTC* z^*B@itXKJmf(>VF1k7p~wXxmS{gn}mgCH0TG2VWrzWPAZ2SBBYf4AODrf`a zi}};%a{txQ6T(XXfd<=?un8cHi^$#+S<=7>l{K##g46g}tt{s@NB;-g<7*!S&2 zjjYxP!&!}U9rj#&!%Optf+5&`Aw!<3XUyyM3m(JCJu+SZ*ozm404t-zY4!<3F z!D&bk{y3SQiz;Im3S7x7;)_ltiy+x3^Cc3#y>OCyPGhPh^4wJ||A{O|+=I0sCM|{> z=+=8{ac%PT-x^kkmCLt;39mm~psnM{$*MPZUFgUDYLv3U6YH%Z3nEbcmVK_xO+JX^=ETRL^ZhmEI*@FJae< z^cnkTbl*3&(eJOMSzAXNv%F-KXrtTXYa$<=d=YAS%}0i#gK))Q^8kZouUK&qV87jG z1XUbR3)>A0loxee+ab0;nIVUFoD z5%|No0ct<`Nq;Ru2Zh}WCN-X6s%|$z4@Q=S>@a(#ZV6uMrq-`9BS;l6WiB#OTg=*{ANzj=Y>RsTBAuZ7U_O|cy5YZ^5`~Az1PmaV zJ(M#q(WAx0N{=xN*Z!~MG{z^@Ax$SEa)l?W!G_pq8sJF4IDf*pJy=mJj0msJg?zaj zN$j^M0XO}G^;x!#&Bp^u$k~IrLhak1hga?;vri`tn*)+khc7UCeYFD5zUs>q38)N; zfWRhTXZ=e9UaSS)tqSb&Ld~Q<{-RQyF#D2ZeJR4dm^UTGM#C_9`Yx|;Zx6^Z^WM<7 zJ%;f<|1@W`S&%jEJpa>+9X8)Bs{d5ndBcRY*fe?kQl90wwKGD*toPE-7d9Hrbiu1C zZyT2*-_it$m;EFE;%w?~2(=6^OerTZ{JUa$1z90QNW18K|E~Q{$0y)w@j*2YJG#98 z8jfl~#M#JHPB1ry&v`GDO`tdUQ^!{(QS*Mg(S%@j^5HpUB%NIv4;QpRfETy_s-K5Q zy_)oqUSB(KGjEPB93-^i zd)yi#e*EHZ9q~}cNHD?DZtZYJ4kZr6dE!NqQ%L>#33J1?1Wd&fVqd(N*y=o5ytar9MUIR^gu1XG?qohPsr z^}C!okV(i#MMYK6Us6`G2BIS4G;JGy!z;rVQV_K#sXB3J>mf>%HwW3qe;b!{xCX$*0^h1KO8?6y^?AF`eKN523|{VXi5SwIj6L z#Re>e9uIL^^n&^SjBoTUr5W#F=Dh2Q$q#ErwG#x|neC!hU`hOQ$fh+QU@Fs8RxDcs-g?s-49$G+~^RD6UuYoS$DEl4W zt~sArhG`9Gc34l+NrIv8L5nW)bB~iND!Q(zY@PZxa>YaYU zp~O2=sxqsD7)nrJ!Uzm>k@h+kf?k_`MK@^qPYm}2;l9cm(;X~voS#tSd`3olOWjgE zeUcyVW=MT!f9bOk;MCVNOJva&OU0>(Op3JDxZ+7GxOSLbdbD*)$fake^3QR!2xJJx zIXbyr120<@aOeySoGO(8W)rn4Q#}NH;Au6s0L*g?ot8%H%~2JCri2VWiLwh(MaT9~tWy=XPW}ML&z~KZKVd zv16I*HzErwTS|-r&;NV`)NwZq0U6ChsZd}&jf;OLSB{U=kjpU_GlNa2skMb~;ZQ;1( zu+_5f&ZIHj+9AEKKm>(FlOaqGE&A)BOqw?@Z2Cgdzj7=qnasBfX>JnLI6u(Re$&Nw z+Tbc%Y`M0~UO4#x1CyyVNZB}>p|)mkD|=Qoclm7;yYUt>Dzj*5vtns|EVByD>D1Br z*5_<%VCgE&`5R>}HOzTC_5m^8elMweWcIycnnu6D#AWbSb+xq#n9S$wC2eB@j&8tu zL@y-&q@2|7X#bkDKGX9GDrK&!!8 zVsWe^RhA*l4XD5@6E~yRG;;eTxa}t)n#!EOsj=ZvM*PaNh2o({O$Y;Y>e+{6Ykhsf zk>M*LN`Bj=gbi#DI_rYJE2!U)PFVMI>Z*skniN3G?U!;X6tGXJ`*wrV;0Es>F~WOk zED5mEq*SRMy)MpY#$r{7;0bB&xJ??|^%AN7L()_XewXT}vsUdHgNmlI?-oP3p`8&` zk1rAYq;gz{jnmgd`0Amli)-msT^A(zur@;m$XPIK5CRatE|vD z*j0&VHf!CIo6r}qM&R`}XbQA^PQKnZ`q6Anpx}s{>q?V`=dZj|Y~5O5tkqS$q3s!t zjz=93YJkGgX*j;OxICwo&vd5IYWm?f%T>$(Sa{Yw_${HY717)Pf3X6<@N3Eafa(m9 z`iC-Iz{^3X@DRs=ppi>ONGutmk;dkh1tuc|OwWb^&LPuG{-vc^G9hC6O;odOb>q(Z z2`ypVy&+qHXs|z@=*RJI%tB+4j~uxYR1mLjct`aR-#y3gN!~p|FCt; zmNFmiz+rl9kHE&?&ZvS?IO=esY#*|Fs>|ePi@QYb7NyDhy+>k5yhtS90 zJb|7D{vw_H4C0GOFjh@1gH&Z%B}u!&)6)&5Pw{^EQh?CCyd(wAiKq z<0n!Fy4hHYzh&%^;2mDVfX0yLi+^D5g0?4SozJ1CGaTxW?5 z00Qqa0mFrY@2*H`sNFfg=C1-{rg@}NjMN7H7(GgHNF~XTHpOXid?2{9{>{o zZVC9pRzq5AP8OviYOSSnm&3H-k#mf-kdNh!66P|TNEA@eQ=m<5v|@HsH(^0MS-4}b zhOtYZ?I{P>OOk@a5FinjE9lWe{8D{iCO`L`(LPH#pw@OAZ=J%*ubQ@0$O*I0r($R< z6ENPaHhc5oK=%4)m((=G!r8m;Bh@NZoRcA`h@?QFo|zjvn5zxt2fvpWBB8j*F`IA; zeQ=Q>bp0tp!#bJ0V2DSgDkZfokou;eCwD(+*}>3APV!`Sx&B6}^25mY*yNJ{aH1cD zIoN=@8F^8nE3xBu`i*{5ztuS$O=C2a>mTSTislMUoEP+NQWe*f-_z%^YLZv6Mh+hk z{WMe*ED{2OlNy!rDZt(-<&Ykk&&4yO!(vt~o%T!n97?fZD^P|n>RPW_vN>X@h=npb zpaIsdu;jk^C~iQ>Np6e_8Y!DR`ED;d0aBxptO=p6R}N_8NI^%}z5DkdO6 zcnD}l$vY=owg{;ykS2}X(AX|nT#Z_HE0CwrOWE=K0kn8uV{mRoU0a3=m@60cx|;Z?7O6GP{mnhJ zYFhA{UQtUJx<;x-iC2?mMs2e6;#QBVZ$%yIc_lZqF<0ut>y6`nd+~+xX%V_T_~>5H zt<~d)mS+9{>M{MVqPdm&sYqEf4A;%#0P7xHw4v_Eu;jU^K4+`O3Wg3`G}|@SyI zq|~cTKl9NG!mepev@nZ9B*EV2K#1MHW^?${i;Thh-uV)~L*b2N_n+M?c_VPoj8->Z zc#igLE?&K0;5ksz&?0smO9k-Q7cPzlebEB8bo?D=9N4zakJk&4Babx~;Y|xrxb*PD zCE0vwK^4I%&mg_eH>Tb<9GX|D$U_%h+t2cegB4KMrw#~8 z>ntgC6|3bg+8-)?rJ*Y2d}{V8AC8^ zC_T42XU82`pIe1ZAN%m=o%1zwokMOlEt@ZNNtJ2uDgyOzfd!&A)r4c{6cAREE8KzG zu<3<8S*vcdet2W^BaIl}HXIvXJRL*}xA=D+9Eu*g*Gx83=ct{_!qXitrNLFb85_-W z@){-K(nabo)I!M-BkGj8a!2ZxF}Q|obWRj3jUm$s&mUl%mC>dh?z=AFnrDSo4gcPU zJs6m78tL0^YW!V5QD30x>vtQr+pviVxTc(v+Nl97y!A%fYF!~x69c!4tc$JRvedGc zsXWwqPsRxZID=VN4QfdD=DO{y?53PR%(`QTZR~zGM%HDKj2yanSyxi`W2Ok2p*!G`x1F1sY=;9=LOA% z?V>zFo9T8lV?wLgv^&LtG>I=51*xSBlRKK6dFEp6{94FmWHKHA;@@X_x2l5%Y&8=c zseC&6oF|QU&{yA&x;0oZ+rci zn81?>k!68v-42Vy67xNne=H0wm|k}Q=PJwUv1#af%Sv}gk$YF+sOX6~c}(32+f#z1 zed4Zy@Suj+mjjJ{t>qmBX|GTQsPT>ObEQO4tmw805Vn+qe@o`_jaKn-K2HbeGNV1dcv&RFQG$CBHY%SzZyQxhW7P&=ro^A^1`~Yjayg!&*&oL z#xwywaf{vMQ__ zu`q#?Ud4gQof=E(b1VMpVf{o6$Wh_xD3j`X+6*&OP-3Cb|GHF7x*?eNP&uX7c5C&I z_vrl*fwtNRnJF_F&S!&5@4m`%+vmIo?oKv+4dZq9cnnr*h{#>Z{Jj0*iE8!P&%E=O zxdlwe12{Aza1--L2>CoO@>^YGyizok>yT=il{BZ-!dJ-C{u>>K$4Gv4;NoAx`Y;Ao zp9s%@1ui@O7V{@yRm9T^&qv*tpjS-}Yx?20W)9dFr+N;Hb=S@_T39yd)$1e(MxIaP zn7Au!eFM8yvP38iu;s5@y#q(-Xuu}~baDw>LczHv`!7TmW4BiF9?RBcEy%d)3>3a( zpp@>c<*G|?2M^Z)z{i{G49aEYpBy3D5HPd>r}+(cEt*gvaykYGKRNrc-7DvJXl)T` zllO!rZ$5sI;l2}A>_PZ0c=n-D`6q?2ro1K1NQs3>&^-wUcq16~ zAm83Z@T;vK+iPzCB~{u}Qj`zPq@y0hwbwX2<@3yN4Ro%Xca2y!XVRwNmHeu);j`%0 zQ4&1)9FxldfbBeEwVw+$D9{pMEsO0e#8B@B@v&F^* z(owc!QnUA!W*Q9c$iiqpsf3_WN2_$@BPPHyXF~qs;$fRgOb*|UFfbE|Z6gf7W zMGSK-P-Fm3tDG%+{?-OqmeW)KML}mHL%7H+ zu9y3ghjeD2C%uzv47vGPLBC_u`b)~=tUi1{ud5t>!AVtAJ#Qy81c{n)X{My{_A_GQ z?xU{(^&I=E+Z{CojYr+HQ-hrcH_U#Q8r~{j;XDrq#}QTfw`>=P%G} z=c|D~FZ)+f>U%c(k8lI@F-VGnpjE(EfA4oRYgvaU(2p5CX;f7gLe)_0o`o~E9Estl zzrJ3-a2AL7(p!97r~+KZmc@7Or~=$+k1%vHNyOhDo8_x?|AlkbGQ zlXM^JkQ=HWiz}8W-EWgmcCp7MP33ti_5T%gVyZ7m`KgB#pVq{{ww%+Y#+?q)Dd-wc z`__kKkyhJ1+QN9P#Zh=4_T!k1W0sl6;)33BU)5ca7Idkd*ZzIc){aGD>wsr73ptFT z6^##!EH|H~WC*i4XKJt%5-3RGwGZ4?dWXU3K|=*X_p=WEu^SB#U`A2AYdAurj$gw#!X@^qEQl$AC=FYEBH;EbKz~sx zF%FjVVQit=Aeozx$6Epc^a=7VP}^o4#M>QsD1JTPT`PIUVVYn@H=B6(A=%OpY&-Rs z@lk2Rx@J0c?HNz^D2|K?Wv zdl>n>1ZX5AmjDN;m!f8F*02nyijb1kvmf}2x?88Pml7U>VXJ(T1>Cs(o^u*K`=FX=A)ha9l}%ECU_Ovm^4^{-N)x0suQCzx%&m+ zQr{=5g#OcsOZVhi{i0i@=wEH?SgqBVLG}z0Q$xKYWilmE?y93$;vblSeI!W9395pF z{gmj7ghJ{ID4$H)Wg?3~L0g!lF{E2EA9yOFbUMbu1$vtYD#Yq_XkGX5Dzw$5t%(8h z+VIohFX|hGI&(4&QR2yy=^!xstn5InkomlbyE+JyCW^MhTPVN>3F_$D25*gwY zzULrw`Z|*Kb8fAQe4L$~4JKTHsy<+F3 zdU}w4r=#;pk?SkvkKQ4qMUk5}C5o`*EVPovV>=)=}Oj0XJt_;`sx4_D*oJNN+i$8-_z zm{yHmNAUjO`_L85b{bWKSBoumU*o6tXJ%S`AhuU1I=t0lLAb~Fcx&LmnNrVzxP@x5 z=c}h-9Bk8XaTI<<$I7!eY`66t+^JtmW39P60s+^R-srIQ>L-e^qowy|h&9dU#}W7nF z-?)XOTL1M$Cz7|&M@wbz-2Rc7^bz~>=IYV!tou}*e85A<^mnYz5zBugk^XQ?LGlRUV#= zv6bPbmBt6d^fiFQN12w7=1PDgz~`?3Q3y}Gs$4>63&4%Smq9H0q`v3ecOGmIeVnxZ}f7Vo%#WG%MPKXB~5H)fAvOqg9E;~d7Mh^Fijel zN9-_McWM54fBk*`>p&|y*%dKw76)+b)&M%6PC^{Pw>?g6Zk{NzoAOzW>PGv;Lw5Zd zg-681^j;`Po|9M0cdwYW-(E=`=ko$k7g(@$6`2n3oR;Fa_^K_)(qM^^prGL6jr*fO zG65_T3UC)UH???QXIJk8Wk6z<%wmEI01-Aeg1E8Yj{TxT6Pn>u3Zn?(A z0ZhaaT}CLkpALlPfJqMdomsu#BeN%q?6lta{V!E{W`Ld$K%j<*t%mOYq4Mm%#A(2+ z{!0@7FP9!rg8vK84Vc65E@&GRfHMH&R2K{e>*+P@cB%qoUiZf<>O8Gt9j`cyqRB;x zU1hJtEX@aPzk=kKg~l5k4rgN{B0QYUMYo`LBk>#4Sz!(;yHhhWGwbW?-@ktc!0Fuc zY5KnS*h+C!=Z2Hq%+uq`P+9ZQO<3Ls9IHK!1!3q0aZ(#XXZI0)2dqq<1;B5{M*#3* z*V0hgnLWFS^Bg3Zs(?5i=ditEx-GHxskBr+;~LM1>3SDH&5VqRiSZc+O(tyg0qkiP zSJwmxdYPQ9W!@FpojMQDTBjaSBMXqosVoNn1r~!*)uZuKcrQInDKn%|sh(#p$`%BU zTXtQ3wa+O3Vns#UkR{Jw0P1FCWhEpe00^m>5QZD}(oT;J*kga^xx{U6)G+wPdp^&V z>M863{4-G!WE+GSEg*Pox}N3C)85+b99BsKBFN*3QPx%$l4Il%#DjB7aob|>FJ<~( z9rfArh$3Dsp2e4Ve^_AP0?YS8gJk52Mm5Ly-33zrz<}j*mDr3qc~z6?DnAQwdAPJn z%CD<+yW2?MGKhd|^7Z;%U0i}c&2GS~IuHPx^Az{VgwWI>9-b+-N_$LsTCnQ#Ki*4g$Pka>SlZ$kcWSVqRZ^=D4k2a?DBx&tiTg;Y&KcK3@y&7 z$NNvceOCJ9h&4vmIZJ284B`jgWO6 zLaI;1_WLE-R|U-WXAQEt7TiIsivcy$H5Y113hjL~NDP90yg`Pz+r+3A`e zI?CVn`CKG^O8sw4qIahGlA$+oO$P9vSU$n$8_HjQg0%lH6B$tAB1ou$YA|r6YFt% zUQ_$XlpBug&P&7Y6k+8Y3wgQ)zlPb$3rJ2sxiin+KOpXvY2vQ&8MKZ!QTL0Yya^$| zn}{l>9{xno;&nt&ogSo2!01hNJ{*IFp0qy7CA|?`LmWx4b+N-2HTUtjWnLJc`R!a{ z@xLZRM+tsWMZ}7woSlWKmTd!KtogbD+eX}ZfJ7p3{*e(&2PAeo zWu-^_XC@I z+$9MVU{_{}=UgltEBw8tJPQZ=%xY(HEOV(&Ah8>RUl&&8i;!WMh#`Z4!vrcZ%#s3c zXi0ljBsV7B9guUN1ow{xy(`P4cz?p2TO%mLo>L3r;G z&4!deWmj&S6oe8${ZVPG6&FB%_)PS_D;g9!Q75?)`2*H!>$eDDj-cY^{HlC+Q(6w8 zRdA_Dz41^$mQ8{yV+|$*I&Agu`XJJ5S__)1w+C2nYAz@8{}j{YLf; zC-nz6jEz{H_cROk*?;#}yw~G3H@7wO!b2eJxOLll)Qt6jYDw82A|%LgY{)Bnygw?t z=8=U=s`S_^bWW-+-chRMsWV+gV~0VscYWi0Y12HFXt`*+(@=e#+@;1o1JZGA0Qj4| z4tbKbmVWHeBi^mS1bzCNHt~UA-<`{Av9~=6ksGjBuS$sBXxmwDpR8dy1xBW7&T476 zPjlge+j)0AUI8&K^>}P}x20Qi1+$(mAka5|U7d@uE|a(}k1DX>>!U|Z(-mRTU4q>=9N9UPil(9p+*hL^NNFNs zZ2B%kEkP_Qz}U$J1ulrRZWSx-qg3mK;>$(2qv$5Ea#MJ>Y%@>Zm(B{b3bU>j1P{1YCkD%BUy25uyD#uPTr&_YnQ8wgl5o+56TmYue#<2G}>F#WeWt2z4b{z+Fib+4?9L#mA9MC#ynRx-X%KkLdcbt`^n6?P zoVPa`xwvBaR)ea7)q((w#5`S?i2P)fO!ADj89R^Q$Lv3+QtcGi{9W6dUcjLbgz7kg zWX8hoOvSfn#`qUBy~;vktYv>T;To4_oeR9808*ZC2xM4$O2(MuP zUk>*e(k)cqjfF}L(1>FGivp)^nSfut8-(Ltu5~b@TCCFoR8{T|AF3p0lTYAs^Lv!s zODHqTjH*){dkuU7R*mO8j82%FwPH(DnHYj~qPc_Nmqn+R-cHsNd{mM`A^AWKyteFM z9))76xLQ4!UNZ*cbFG#fU~7MAsHYmNpXk=VGA;2_7|6*qYs!Q@)9B6C`n;*o3%7R4 z%WjA~H#0tOq1sDX!U0EEc+Nb?#yAX%bNOA-xTWzcnJiK3eyk;=rj=>kPG|c?p|k7q zm0iC&1H4lu{tGUcfK1hU{-gi`}CF-IA)`yW()7f7(jxJZr2Qa&$SZ(^MU%l}niOrlffA^xg*Q z#AfAWh?0y(39UI|h-I>z8{rnTzF3e7B)OSQ46Xfa>7`AFOC(6pDXHTCUCCdosh;8$ z`X{-Smhj3InWK%E{(OpWmr@wibU{?!wfX{!1_K}R))|9VI~5L?|tBdygoh}N?Y z2?0^_JHxu>sO$wk7<`Mor=vBD!pC3q`e)$J0eVnYTAJr41xQ!VOw+-}z6T1EXFJ^H zoqzjvh=+J?!P~-LNgMF+P(5^no!ziW(S8y1*yd8~?3FvPSdaY-{##5`G+`_&6n(@? zxVh}mFbQJwlFq2y#d9^gFH3UE!hdthA*Tt3_s`9qE`0#ZE!zc5{Unz)paY-%oIZAM zwT^LD()!VTVpDExA#!c2?Rv7ak8AdYFV?I(Nh6lAu+6W4=ga$wwb;SD;07f{xj2@e zI~UgD^>Au23J+B8Oti9&gQbsfg<>Cx^nLoH$+d&(5wF4W??SY%uZ9ebyASJu0ZxcUpS_OA37w4G&>_%S zH*9u312gQB+CeTLt3qr#J4ph)`bbROTk97}b+p-Yfp=%~J$s=r=wI5$;1u zn!&NVad}ZEUeiAOMR$|!s;se51=w)H@l8?w1-?UvFYX1h%li!TG$$9Go<2@rG0b~n ztkRT1i~TiUZoWLi8kPYSGbXHc-W+xWD!c(V<`t zKiE6Xa_=R1W_iJ)uUlon!f_#WOzky7>6#UU1%dTRx^+BiztE#3A;AqbdG17*7|CcE zx6fGYG;VS=GK+c}*r%Q~fYavjQtH=-UJk(*_xqyx!-1IW>XS$9*ENW58Hc(I!Tk^L zh~c5buE`9KRsO9e62t)ndfT<)fu@`&D@*#Z&~5wCb}!gH3+s!}PWHOljtLZkzlUaE zUf#E8G!|A}8WyFG-Cb(ax9NNE16fcoIYBmVj0(R8P+C0uHcB?kF1)|2NFOyaYRW5L z9!xyv6&P9L2B5Tt*$62GvEsDeJT1213>6*o3~Biaa0ELy%k+i-;shYCfE-JgaLP*P zsC}y)z?mxJ4+Z%KwV&_jwJ}SZsYHeLci8r|Nq0AWR|KMfdoK%_Bjy&4Nn>`-aL1Mj z+leYu&%R0uT3)e|THy%uE-d@4^H7ntcpM(8Ee)U5(&UWBEHT>p zNhAO9f_9Y4jTFxrHADAJXlW_XNx5UGSNmE|@h}_x#M1bA>D>TQ!5KhsZFq#nkcU`x z#q)3n`pgw1KI^8&mLZrL*reqhq~j>bZQsV{)<;fMkj=fzuDTrb11Yg<^8O_J1|U{f zcap{P9Vc~2dt$fKpi+@vww}{uLRr1=8)F-4Qnx!bMb|%X)RkK`|3cC<5bdp!Yno$} zv6?+s`o4OE`mItAhLQ`+LA~xc-tCUp8oTOZ^wi+_q5k=bCX4Ek^S}*qXC9vLPMJK7 z3;2q3Z_l&I^I_(g+!pb!35**8%#`Q?Hd1N`czmh?Jy4k=Q@kWDnDU^DGYWX$zG_gp z=Is!qjVu> zga)7rG*8ij$@e7Qzg)?j1;R2sWC?`JP^9^R6r~ zr@9EKU3p!n2nlCt5}D<@6=ISKTM`Z4UnxXOeDJwI3DcUNmH2v;+OVMO%e}nQ4kXyx zon!@bs@>iwTgj_EXh)W%c+OP0!MW;jmOu32#SFli4-6V69}WcF@jjSt>y)hAPY!F0 z84a~rkn9Bdk(U*>$U^gw0vnM&^^E5B7qZ7i=WO2*&Q3BsT!pUaJOtSGwdGLOrWii(?7r--)n>t*PMEAzR%X8(eE%ZCGx`C!ppTB_qbI zC$9pE^Dz}&@0rmU8;N^{nbcIWc}c!XIU^)4-O?}qAu20VuK-JzY?kE>#4I zjs$+5+9=-s`0P4etFT6*|j|dxI_D-%^{Q zB(t|~qKS0A5M!SGVr_fL=Nd*-W7Vr=?Y_KrHUszR!eXk|ARVH(!Q(@#xNo!u&P zfot*cWXXqQHj8`>vnnrbj#Zc$5Bum0O0%3J`~V6)Ku`Rp@mM(sID}>P;)Six0%I=_ zoA@Y4%}En4 z%JYY+SAJE|GwR7#gRkWb1ILGf8qvzdwtH4Jfq?%fXS_0Ae)04b7UKyjclHV}SdWPH z=|raP3bdSO7pp(36Qbyz_l{0nadX+s>!bC3(hA8n2*dPEZEkOG-!~jH+<|_`1c7^U0O&Vug1$BX+K{}Fwzjrkk2>Hh*~4DXCpkXrU_sPS#kLyHy@Q3) z*#ZF2XH7GiDt?7~)@o&(qR+#FG+gB!0Qw-`YR4xgOw3kiW{d&#L7M#iDG<7U<2MAK zmjrfHB^hXn0H^E!GvL2p{Z|_Q)s6o@%|R}Z>JzuKVZuY;Ckft600NVqQ=C_cPa_;q zL+8f!15X}^@X^(nY+07-K;FH~2JJTD8o+)6N1);yteL_pC$TOpkC|LjkCG zuaqUm71|^ko+{Yg+}-F0O#*${F2UWc-8FE>YTOQZ0{C;Q<%bZs>m^Ibz0JL@C;#_5 zH~xm6p5`Lt#cz>m5kRfH(@+coozh6(M)b8r@ow*I5 z?)mqaX@B7KA)(H;J%pvD0Lfhv!be;zDQD#`AMI}y>RAeGgBfq{%hvdNZ!Q8y7;gSr zd!vyq^7+#EC6DFBq2yAczqw||4G&)I^I}h5%E&zDex+W~b{4 znjF;t`a=YfzrVjI!CibB0W{BvV$0*bjiUYg(&zZT9RiNP|9dZuR@I&-&i)p?yr{z! z!~-LR{xecb5&N@U5O{PELsKU_CB#@46tyg&{m*M_l`VZ88%{3GmAp$}i5*RI(UnI6 z9?TVRF29u5_Tt6K&)AvSwMl@*PD!3k*1yCxy$Pi3Xrq6x@J^E4vE@((yvW8__*B7h z^M8Ex%{%8>dkCNy{BjHRA45L^42qu(iHJMed*NmqpnBE#sAZumY4h8A%klruax{yd z80tnX_rd`8`akB8P$pZ@*a=CC7<*Zo8gWto$Z`znpO@no6I(WXu-&`2fUN(Q@OCdr zEE_TyfbiDW!)_mibnb(9d*n<5so{phztuUu=i;NpQad8yC}gKGb^lFP!9N6W9%7jo z5tz#CNN*{(yufY#=QIa^ zJMI1Uci?o%|NM4a!id1KJXIi$kGSdT{f||*&Mq)MhRh_EHi|-Y7*8WsTHu!BceQ8# zGxuF^G`s)Q=Zx93v7=FcZ2e|P`NLzd`B787qFv7DqpO{lE@EEK-ACr^6FW?$eIY(e zJ^kgX;U;F63>04Z$@fP!r5CnpNOq%tfo-HsFhS2hdS3aFCUz!=Y^fClb ziP0MGFFAFNj@on14V;TXgvG2VF>+3lWLJ*-<02;l8XVT5kr3DI5UF|DQ*5ar!PfdR z#A}HP>G%)-4}0$!)>PL0f$BIT{}~lV8O4GyHUbt9L6I6eQX`@uQlcnCq$xcF5@&oB z5f!CK=O`@*LPUDk0Ys!k5&{z1NDB#s2qZufPI7m|0{-vEd+&4abKkFUvd`XYt-bbI zzx7-DtP49Xj&KN6d?a+G>{EijCW1$;>R401_LA_VzAnBF&Ncv)uQPk<=3U>31DVdvJFO$%Os@c}Bi z(BB|ZWRP{I8wUoUrU?G>0*kxNga$fW9k$}`ESS~aEX79_3)H{&9?S4@)B;hgOcku& zuw&tTGV#Ywg#LGk&${`qEWX?6e`WFitSp*VWr6=ro3gRH0Dp0y-lKQNI96IMI3Q0Lg+l%v#Rc>HIkU)b+Mbp;9hMb zv$G(o^IH_Iuzv63!!>@CT8qC2wYweF$JJZ645JKr%U;!d_19zhg3$O3s=?q=gGU8R zn91vZj?E1_FU!7lNaq09CCwqqx4+e^DR+4s)CPqSj&#Pw=?mS zF6^X`(Cqt0bs3?b$?ioY8~CV?>z{y^yim+iR1N{&KDMUv46;MmTf7Taw#pp~ z&L7AQ@}-INdaqS6NPf9^M=MAxB*OP)`rW%2-z=yk9~s%v{Myx(AsZUvfZp8Nj%s=; zt_>Im^HAQHBH(|5q$dNru3JrP8II|7Qa_NBSy5I*%1$iwertQgO(9PY^31^-D2kX+ z7D%(co}8UWd6CneQ!UhlXm}4csn|L zezH}-^ZpQZqpLAKszrX1xNaBAr`Q5H@u+U!Ow4xDlo=W9>Q4^5WjjC=u+n?>6%9rA zHt&mcv_?)(d+Lh|Q5AWDhH{uK_%r@*Wwl9of-9QqqM`1|ha!ewE3xBhabsF4c~Bn- zYFBN@(C$tbg#L2H8rk}9ts}QDfKkftKIF0|oiL2>IDd~!l2#7G)UmEsQ%jImE@CFp zTG0--qh4M?OYTOP5b1g1{E$n#6%x|QLF1B&Wy2ofGiD0?jDXsH{<@<+)>tF4Qo@N~ zNQv*Ru@i&i$<~=|59hpXs5~^jpYDB!Uz`oiw)hA5#8lnv`Ms)iI z`ZPn8Mt1RRBX?y}XSxp-D~C=-b!9_#7Iioy63xasugnXIz!!C3la0z}w6C-vdTi_` zu83wGqy2^>AL~^c6AJy?`5D?EmI)vU{)&)9K-xk7vKTztlY7cm0@-eL+brlqVaufl zj`aqDDA27u+vP#fu|H7`C>B4L#<-BTqo`^gDV#g}W0xXgwu{}=*cVB^V{v_{c*onV z!Fs(xO&wj?x0QR(>s`n3B85?A9L@Ht<+5(o)RW~i*q*ZWc0t$J+JoG%gl@&U>_Pru z106C6x|SrF)Bk+iSFjawMt6#uLnAYJyTXod&i7k7DI#E$N*(XS)dtZkQRb#`CjY4! zt!wVexU7ApEJ8-id{UjLjg@wJIif-@HQ{-ed9K<-X)0~2mRxb z6QZ)a4HOuK7n9VTDgqfd8xzbbM(72$38L5}?i#kCR^{vNtwRku8o;O)8y7}Vgva2^ z3I#?o>BX`#Jb$LHiyEf91(-}1gX*C#NP25+jmQ>NX#blj=IVu9!3ezX7<$^n$6DX# z;`>|Hx8^^)7>cP28$Q5dx3drq{cmpFD~{~oNA`(b9kQ7xnzp=jxdj!q{Hee^gM^Bt zH*~$+PHZO1az+Gb-EIB+Mj00PLNJA;0R+smck7*NViTD7V4waOHcdXX2J^)QW zc*WW-C(jP;b5P`Y4+MD?dksXn$~MZ^2jwKq`=90>zqZdK7f)YBAMnL|azW_F7vDE*-ajxEHq2rSBuKAms-%77Ss>B& zNct`Z4@+}f;UE$-W;l$icJa78dsQ&w& zDhZQEJ+;9070ezKG`=@aMqxxZ>5tES4Cx?!T05#eV;G*WtK1#s5Yh7c)59w?l=EIo zQida66;?uGA!7YieEsZk2%0T&J>s5x3;cI z4@vnubUH05;OU%s@gM3(!BM#v(P&tLO#H+jNvfCX z-Izc5J;O`;XS#u(D z_i3Fm!b0;ou=AGI;X(3<`{>VPsmj(vT2{!sIk@f$gFS442Vfll3*tMHDp= zH6s0ZQv%jtGxnlaB_*zgk&q2hL*zk};afF0_r5FJLFsXt{!rH{iJ!_)b2mwf8EM*1 zMyyk*?(FG$Jl)PK(PAOUgV(*|8C^~@w>HFoxB#SK`7Njkddosg@@XaaZe}Dg+cuOB zuA`yGtPNG(6RZ&G*V}$qpV%X@VoN?S3l$%F^eKe8Q?5BUFNLjmhS#M@+&!Xy7F29~ zDM(>IvDUzycg&^P-{57jWS*newj8}Sqj;1;EHxKzZY$?c2%in5_t)bpA>Q=L(I;()+vfp#UnsiVHRu(G}C zeOC?*rFXqCdr+lI(G^!?qU9((@jhcMo~|3?jVXVpPyZ-=YhZNv52R;8&3pPx_QF#L zM0YOY(%=i`1;{*N4|H&iOl4Ks#y}5QKQK`oW63x;P=V)=x>{%23mk2JumM&{-Nr+| z*F*o_E)*3jpsR3z59!rpG1xS9be__b<8yoZ9vXSH{korvfh)4VxC6zdA~T*>wiN%N zioL(tPLU^z4TgP?wnfF~Gz^>5+drQ|-wan0x|)+7WWLj7<(WV$o|rodxL5u-!R`^7 zm*!UrjKVYg{|G;^|61VLoptv=zz^PSb4Z~&`D8&AE0g$%2i%YQY-n9QsJhKzYlnP@ z*{(KhHFkqn!&=$mzLqX21>V#&>(6J?39wJk*|J{Xhqtnam01xd^|U-Br=wNOikOpa z>g-zi$-7rk?a8QkC4a)i(?KHRRs%=Jb2~e3kv4N_3y@9v0lbzA@udLBK06rZth!= zi-x9(nmh6Lh7}vaGlu30HrW2L;&;yjGtz98DaxCFg76{p_D5_sDUiaX^NL(v?xCSH zH?jv_2uAg)>jR?gB6;W(WF(n%AR>q$@-g6?MWXjx%QsFLadNA-Gmm zz7>*aCVc3ym2JYPoVb~NSYP`(s&^BC+nNe87rCE_x@SN(q_K|i;7_wSztYrqp}GU7 zUgD|1PE-8vIBXl1?fdDHsPxYfH5SAA(%+L&Y{nbhJ!^5n{!=>~b~p#rvk-T0Z_0+I zy)w9cCr#H;-E8i4G+#vX;WuT10+qKqqZio`i3eN_f=^)2)|zgeAm7YzbnP&OS%}Bm z^><`6K$jRd!_xL>w0mRJ7x~)pVu7~%bUIoKv)UXtpn|Q}<$N^kZ(W};aaN-w1DPFb zOW%UhWt8eO)d={y*ynJ{1i2Q04;(_vh zi(5f>-vDcc_p2`*HE`d&wT%#nP>1$V>vgYII|lhSl3p@H+5BRJD|x5d)ZM_jqfgp2|!-TK5QtIJRJqo@NyN&srbuL}2 zz1_E;o0(Dx{XO~9w>*M!V!ebx z=Ki+Bg{U#cn_rcpM7F(7$5+MIOs_3;!i|!zA{PRb&d*sG6F(KlouRzF1CsFI3%I({ z@&a$$>7zi4-;hZQ_|CmG3txW*xi$3|Z`s~GU!vsowD7me3kN^J+qTd6ze7lX`1Svl z#kZowe`WE#K=WT&{NF4KZ3Aq3FW*!k%-lNpJIG-h_U)p)S@)!w38|*x3pZd0<-2o}LsiHQYQjH7pF(*CWXEYM+*gm8TAVL2EuZ=8ectMW@+QMIul)(oZ^C`Oidy z212k@Iqpln?|$m=nQiRNtz5DMGX|IX9vhEX&!O^tO%kSOgbFQG|xQQ{dH1_u| z6u)%uFZ1iTQ6{OF4h&P=n2njs?kxHDulWTmVNr7N__>Mt@tP&Cg1@BhZQwLqf_pNL z_-2Bd>XPY0^Wr-Nf`cVvf6cG-i7H52J}jG`{2_aA;=Pdi`ZjQ;UXYo$h9$n>aD1ik z^?jhAcSHZbFZ!?OfCO0oe>R3!de5!0xtO^SMf@GeBg$_ZEHyJbX!8zu>h5a4eOf0f z0C3%x3K#+7o^ttsN+8*9H<-r4Ocw&LzrPL8$QomQ*F%u%x zPELKDo=a&(1?aLz5sC*$TA!XsUVyF&!d}1rQT~uO(DvswxY(%1BrF&DNZYS|WSb_$ z(dL{tLwdNGar>F#M-2ec{5;wJ*xA`}rG&LDSLyrVMo2V7B8YC@>DFP!D8oC4(Am4; z-pU|8MJv7Y>+;S3fqqb7k3GsrqKQS5R7KH1pD0g{559tRX%Oa%ncdgA+>1RclJMcD zDQaf~_M_oTjQipfc~}MDyd++4?^>ps9|o-nPl1;mrCSDzXUO;#zL-CBLO4y=?eXhk z7Z#JojjF8(d}?(8VW9s7DL+4d+*3(5?aShfP7+c&rVWtcHg|%pSC`h$!TE&UC>=IR z@UGz8?v-7{rv7*c#s3ZpjCdNx_Yia<6n2R_31nao7_ z`%CB=`XG}{(h#L_%zOG!87BZfD{<6AE7b3t@ijr``i}P8Y!pmd}c=ny9@BB6{EeCcANHmuQq!dh_ zf21IPMNhVn$yAn(0d}1)syyMs?$)2vH!;Izy-F)eO3vk2^^j#UP$0jv zI)lug*EqyZ;cA14^-bv<#VH&690D()C+ubdTFP+d9E()8UEOtPz$6+`sdvM zT3{z9$HsP7(F*z^@ z+bM8N3#Sn0+*EE*LK3Q;6Jf?UM#l09;QJs>z)ex}5BF+Sq7qw6D=}*UuSg|WDMm6g z?vPv$d6Lyl2A0&bXUSb6_|2{?$Yx#zD6RnA)p3BpF1C zT}z|P7K-m#g`l%dxP7^>=})yPzCrHKeV8#B01bQ}GA2G70;bQM*5Vn!3Wqad3#AC@ zYU98NR8>{Wx_*u_UNl~*pwq_zBZY0=O%hF8R>PtB1|3Ds8uozdl$Roi+ab}}vm!xh zNy!zCp;I8lA3};bVt-3v{<+<%YHH=1q@oNL4VW)5;`p$6)jWeK^J>^A0rn~#^H4mc zuu!aL&cSvL4!M<4 zx)X%!#Mx(i;7CzgPReaI#5@i5-dY-Y_vdHM(f~)V9og2Xf@hhEUy$k?TYDUztY0(^ zc!pl=smSteTghj}{N#1X`C3pny{0NCns{!}%!oiBs^EV8D(a)`QW?|p_T*~OG+5AS z%Kc|@hmvZl`n!Vs%l`t2&;m>4ok7AY%7`QnSmy!~E8-RVlSK(J>s6*kQVkXHpI(Eg z99DY;JF{OwSn;~Oi`A7T0H2qH^XKmEA4F7+fAo~0{#I>5u_iNcNR!s=YS^dk3{piS z6JIs9YR+hrk>LJ_hyJN0KtTu9S(pro15A%#VSo<@oEdVT+E3Gw)&wE5M#kXa!P({< z$rPAV8`A~YUinYHV6#PCG@MBmy@I$&_`>tP;HJQYh1>ONCLY*HSGnoM=Ub5I1HMPk zcexaj;5e_@u9`&A&rHlaz=ZzAa#!Fg zXNQGmSm1O3BCIpV{`mT}C&g%p@^~cIAtkEy7mOeH`I5w&e$(hGp!M)hEJ;0c%Gnh$UVcwA!rIds#vp@ zv_`Fs;KTJXpH9TfQ9S(oHU74cSnORE=CMR>PSZ5<)xw?ZD}!YkqVNffeiti5Q~l6{A&&Fd#F4va&Uk98}8<=bzubu4m7$C(;+Xl%^_yHWCTF zZ*@RRlkgNSQ2pWf*-UMJUt@vQ#tt<2W+vld2GRig*kC+1Ba&NY*gQmTmnMha-9Fr1 zU4jd=DnmMG-d>%)i7j{6HB$eoDyoRN!TOy%K3BhJmqWF&Nb>lD_L@hU8uCGojf{>f-QeU2T zn|$$Fo^g~KnWR{`cG=%4oq#u^^cK+{`cjDw73g!b<3N!lra2QjH+k@%@fW50f-8=h zR~)M_4lL+Xy&qkW$iBb5@>W<_*!Y{DN7IsCadCO)c2huUZuG{&^LZbKh4DfN_+A%y zvIuI!EfeE{J+8Z=^_*5V+2{&XtTkL4lR%k3ffor?P4A#wRu75Ikc@xu@g9?Gb@-}J zG2$F1?_Am{CHATeNTy@yBCj&vN zkplfR9MriHl)l@IW{5IQRMQitINeas=^pm4?M(TzWkt)%K)!6?{^vrs*;Fp^!kO#XvWVg<7$q0&RC{Bs17=0yg5t<+m_TO?EPHpz zEVF0no|wjRpF<1XSYt&tR6wF>cc8S7*JN~;2|BL}HbsKJ8G(}XAi_g&Qnq1P*j=3- zGIdyI#wKuEajyw+C#azBc4th4D6>_2%a$#4os#iHsy2j>>VlOyUboHrP&m!^VZ)|P z#b8gR;tl5hy~(~Au4Do0`8`teql$I&jqWYeDeGP8iz)sKm}263o`$6)gaRr;1Z!sl z=hO*>}Rw-zEA7=+Npn^ru%hr4&io;KOkAVU7?IBXw|Fxa{Xsb8ke0# zN@9WFT5OB@MA-0Yv^Ls7v4YBRDmsHKUeA-YeC++o8VE8wyV^dKOYL9QV^zZ{tphy)e@=vOOPDWX8l8ks5y0|-zsAS_ z=4rpn(J7wYz_dP6{C1#TY&~~x%u6?Tj|oyJK3Y{>gFe6fERy7>3%=bE;XUAXU%(u3 zv)&31ZvF?yP1D|DO`itX5{O161bn=o(kb}P*tV~+SV1<_qUs^Q2s5+0LT^?-zHi$X zFf-m$ms2rzA7*ks#XLStL*WGVT4#|`*2IoZ&CVDz_>sX(?90A?0U@MHejf|72Ji%n zKr&UZwkL2Zbq!ptQM``uAL{+t@|l5fI@OR8T zg~4F&2dv|qce?Oq5oPia4OQ;mNogE>efUvNHpIU#z;@eB<(6N3?aSf#-6EwrYwgoA zaIun;lbeXqa8A~4&i{TM@32PZr6zDVW4@+m#NszIU)0Vv2iKJ|MfJ_c`!2oI3_ktS z)(Fu2y2R+%;WLI*b1YG-j8U@fkox2i@8CpLbD7EKBZh}BAEt2-U8nkbrdvVLc z4?iU+rFz}HZHB!AwpRPK0>cTQZ4WILF>_fJy{jQ|_%2=If+cR?4T7!Qn>*34FlCAXUWBvjqtAUE7EI%mY2j--XI5Fd8WLW4)`8>4fK2XG~PWan`pu z+x4|jHf~_9;QU3L19&xe6J#&Vq?<3?^pZ2AsvU%GU4SNqAPaE~Okl44 zjxK6xQA}#3a^w%=^pbRkNBJFn)3!^V?R_xqxMq=E7-vBu->Sqho2s#-fB8n<>#iB% zEEXN!XHLUuN&J>fXxlQv<(`KS_~!0-`?6nv*ZLlK?tF%i01hXB821-4ny4|z$mH+Y zI~cNWp@hp*@lvRM7Ff$iwaJV?=U|VBnk_xnF zVi=w`7F!C;^nS5_dI3_yyn06a>H!?OFYf0>v$KIKJf(#IOKI0;hEHR1K`IUQg>GAz z(IPcu<=`W@cp(CA6*Z|P1S%3|FCJMmD{o$88c5Bi8JXIn`L6ipZSW{|^M0bA!|rwF z6E*)?WL(kKNKSDP5+r>^=pAp*NH7j&aO|pu^UB2cdU_Bdz@EGIspAic(TAgweUavs zGLeh2flU070wh1oKH(#h`Yz+OfnoEB+?k6h3#SIc7)lq&2ii;faT!Z?N&*8B$Ti!g zr;YmB1}a(t6_pm{{}yloKvTKF9@N)pL+f829(`rp_?W^N$(k|3wd6ADd={GCaUM1A zd=4mJ9oRf4K{%KsMk9dvQYdcRPG^-2|m(Z>v}rcI&l4)9r;8ks|2yC_E3s-2$&7#h)G@SKL06Jbt(L zUn5IFZH!mC3_yOrTRwQneUPGF`uRP`stX<;SoM>m-9+o|bt}?k0Hn4E;PLb6JehqZ z@7!R@#hF&+?s7zPUI6pv!8!It!8$zX6+E94Wze)Jr8zNVxAHa%qktnFKF|Md?GK&~b4B%ss+AZz{)UtI%0 z>JRZ3y1TD`SO9n~RbI3u0kZ!24-4kIWN1-DAOU?Brs$xm_hb=L^YstkU?>1lM@gX0 z1z6(zAHIdkU)TN}@cVzWFq*ay#A6*?#pr#PzJOEnONJU408QtS3A1C7ed%*UDDh!p&x`BhA4JVAEWH&sW3B3}eB=;QvmpuI3s7;yApv69sq#ASka)}ydB9)ik z-_Y*Q3ifEBvxrC=j!jPu$xmA;TfbSAIs=sq9ED@f3A@dhNdFk(3{jrTjjQ{HeSa3A z7AAl<1=dJUM;YzX=y{Md=OfHV;pJ?+_YGGQ){%|Fhx4TY-VQNa(DfU5)ZGsFQrE_! z>8HcDDCH-;A55*Q%l3iF4>a#f!@j|313!QdjQ>dH6o{j;5yLx-8uH(bZf{N#M*a2` zOPyDPxA$!;Btg78rzXg2!t$DXQ1`s~FP{D7DBY0-SwWOsoI)^Hc}K@C^(trhZ9G`_RJis`Hlzs! z1m4?u7dvdttf<96a~{nmc9q7-CHkN_jI!^!4YE06(1oI(XV6t=Iis@n*do(m40q=U zLQ~^XhVKwEvM2hocf!d*IKPit%Adb@G#`78<_84?#y*2=?8*D12=d-{Noc9I0z?XrH|M$Xyx~^R z;KlXsiqNvmW&6{KlYFPy(_*HzMuV(BI@cQZfow1SpGuYIM7SZq-b;Y&vHNo5dhCho*RMZVz^h{$56r94MvQ{+alcXH4_=D2*k9 zip{(t%G|(R>+po^>qCPM8Mg4)0T!dCFlPK}x;;FnGsI&d_R!D?t~5`hGiM(8@u)5& z!HQG%|7&Uv;L5CsU28Bl9>sd>K_#bPUj9@jS;LA%k8FIA z;SW}-cbOOI9PRwU!&DHu`cNr~X@$Ib0(NtVFr$&tBYd|7 z;vl;ii36i}6STEXjdID%mhMKMY)EQ7k81GmE^DMz^HQ|9$1dD0#%>ec8yE4%(E0@;ObbsdMZkRXg9l@5 z0WA03&Rtb$ftYt7uwHgg0O(Z~oj#-v1LZ6T6n4LsvgUOlV+q%K5R%;*uZ9o&<8_mo zgI5S>Lj_t1NZ5)QvH`zm6(`sdR3HA=f`fpdgi}}1S~lgHV1jO&d6D&F-KSu;jleH5 zfPYX$7E~4*$r_f)l>XBJ>cn5mbrLt;ofz^A-;2KAZi61N+i`oY_Hsgo{3)0_41q6O zhsM=r_XRXr;|;6js~;<#hPgi+hzV1NXk=ix5@8Eyre%m82nV@ZI*^<}s|cmh-EOiu z72i9vT*NIWo{ZAXdyJ1WausDk_#cl60F_xco1fX)$~>kU8KH2uU8#A?IN35IEksh5 zcTyiu7j*7p5%)uShuctjnRx|F=7j3j9xp>4^+#`Go-HVyxI>J^*RvSG8mE;uM$X1% zcp{3m#a;5#Yjcd74h+vtMU0(-YZHslB5e)fW0TLfCLK3(RYFEN(;$L^EVogIsdhB@ zM+I5aPz4$@w(#5~Y=qQsfnjqkl1ey1c;*%FBnb%WV08vMD>0)lv@yUv3qQ$)e|cM) zCjcWk&?{as3)?$%ax9Q879A){+r?WY-$b%VKwQ|dLot#D4R_d`-HsLP-*43) zX+(H14(nrIj)YQx4wlAZRjyhio(DbEd*M9<`xHnAeg@GmV?AMaoz zw@a(PU|(NCmn6!)a{hk!*YSfe&9c8uyDPcGr6^&%PxL-RMJVGQJ4vT_5FBwY*_Kvo zz9010usz{=FF;w(;bRZ^rtcY~)A2p0ff0VGXe(w#hb4o&xX$rW)w^Puu!mC3lA;!r zRM0|&o&Z8fI_mJM0!fj0K^*<`fQ5RRC);Hpd=C=-Ev>dgFgKK}f6`Sqhp&7SBrhO2 z;FgZuXkUn4hp|EeC51_=>>;lWn}XcFQr}P*o+iZj; zk1vhFtHzl8atffS-%@H2CQq%~1{iS1W3R|(O>VM zJ=m|=UANkS*~OoxAm|~WmMjGM78+(FIteK&%kq{R6P{2Mv*YJ~wIEC$M#xMz^tI`F z4NyI|UVdHyYHTAn%AodWZp==$$64Ju$s`xV+0MfNJkF)}a)QD2A%gV-mR`vOn4R&H zvH=y~-?{%x|F|f`^%D#{Q}!M=?$m*`hfts=cy~-Aovyez501T0yIMFsXrTVVkf~8x zyX8U>N~1QtFZ(LthxB9W_ewH-!r;NDc%K8S#LseINdfO`wGXIq!v~5`FPMM4+U%i= z6ZI36_4!Q;a-f>FbbT?X>-O%FmY$5aEGj?(k?$`+E<_Z8E?it7LaAMmf2c+!dG2Kq zYy>z=9X8&# zTZFu2DQ096GOfq>GrB*;xdIir3u>p9U(=kasRFNCkiK2K+pfSGYHKNUVjB^L0}Is_ zXN)z2X=>d`*@5X@<`b3|1tG!d)y`pcj*kiNei-I_dvXtdb}()Wk`%vlJ5fBcJAe8G z$tIa&vp8|vy9eOhNn4PH(!8q?R!7cjFgvunv7igoj%_LU@sd?%0(so>AdvKcda!YJI`Xw$WP1=O+Doc{85-9Y0#Cp&er2LR zjE)%Gp3?(kNPfZpbW?^xI!gF^51=SNPF_o{#Qo|UGsr>=9nOY+l2IN745^{?AH{Wq zGEDCjR>l+!0Z33mG`)9dUAj@cf&MQ284!z)J6-fh^5aEXnG^oTHG~Z}y;C_FVyJ(m z|3Yy~k$V1L5-a=0NC1<1CLzbJ1a?^pJvK@C8d9<%?+-TPvD7v5I_;+h{p~SJI zAuny>=ie?!iTAFBUCa79ZdDFzu`na$AM}~}8UFDiyL&=sAr=pxM(24RAXdBl_`(r+ zc63n}Zo}l1F)!b{za(p4He_wKhvu=IN{^*h#Vv*!O#c}g8gk6L!ude#n%OtVp!EP| zz;;jwIJnIVqdk?x<}xY{z=&Z+RS5{jAPhl?ZIjasK3nKKu?7%KhHPA^(BwE18sh^X z#9h1RzDxWM)*kt=>axE%*Bfn!t1PqSO##x({P5eyLw7wQqKiv1&Q|X1vT05SujKv=ML5SMCs;A*bph9XuQD_bKhKOhH};nP z(R$)Y(l0G2J7JV=2BE3LF)y|m+P~2KjO7Sy_Lt54;{gyZ9UO1w!)8!L3YT)|hF!eJ z6U=BR?&e3<){K)6-Vl>wFPOxw-EZs>rh_siDMUBlr=j9^pC`7M(zQ~Suqpz_v%Gkb z=wS9)cB=im74p<#wECM>IreYMx5sBxB&E&lOX$&9xWy#K_L;C2?VM1xC0!1jD*4*8 zf9sDzBMb%Go5Y-5L7P`q><-l22gIVsD0tO+Gp!HY&wb~q7JJ_iI3vV69QK>UVthZ` zd#`;p@V;hndW50^aOU=kw)XJakYDGBr;ss&ZCPZYZPM(FHzCjqG zSvk3dGL2U5_1b+A&jk%n#17I_v_s&5m;mbqezg(wA(c)T>Ds%9T!`w;dX>h4tTe&L zbpnJyyR6r&TOA8h0)aEmL!Kw={b2cav#GQVq(5%@Om49#-4S{qO(SyT9$j|OLL2n? zOD;rULZ#P?nDSj~r(CpjUFEjT*kPN@)AVhK7w$S{!uSE1Be`(`HJ_N;xv3NTh*WqH z&NXum_Sp!uGW+GdSTcO=Pc3~>^Nvsn2;}!eT_Gfs3cR*|6T4gnNuAM8fVOdm>{IFkGZPwc^ zqrhlx@fC%TcnNRe>p43#P9tZ$Ay&qKDnPe6y?{<5*U$Ld%)AAeRkzRcV2^T8w{E+h zCIs<2KA0X^S+a4``vp($m|8q_qSjO})Di~T23@2TwJ^8G$y=0g^p$_!?O>4xK~35Z zTTnE^G~KYG!s+GH(Bk*KM-!UklFu$6?#0+z13mD^%Q=*mv+)fXZxFTCfT&gc&@XTx zNHcFwj{A=B2+%@b9xlLAHnmXA18>%f8*E4)c5~w-!J~XK6 z@Ybsan>MoXK_S^48ZH%&(E0KQ5dAhd@wE1!gR>8bYer9)7u%X!=J?sIT0W%vN1+8CMCK^L_z-_e{^ zS(p|_o`3JRaHb0geIT-Gq&D=(^8E(xS&AMA`kRU^Jvd%IWP$98(JtN@(jl`-ILo>^ zqBXtkv%m2+3f<+NF2~+J{GzD4(*aI1R759vDrbaNufM)c?A8YnyTJ>G_`I+na3T+y ze`QuVfhO`0pLEiGKq?)-kGexL?m)|XTf-79E$jeZNH(7G+h?9LHKSJ-ZD&`2 zUk3Ch4Eq9NfQ&kgqYgH&(0qn6dl?mP!mc&>Pf(YnMt|-A9XHtkJ4nx+%@#1*RLi$p zYXFOY8M#R%`771MCEmhv^YGygmNk-`06MfiMqX^NjcUJqZ^=6sMRYqsVm??=c*0YC z=8j}Ld#mT(sZ(UB96SY`%nbqUCjRg$w+pbz^iB6q-G!jl6TAo_siG)w`)t8rfp19J?NrG*+tdL3T3-Zc#)jG zstas9v8SSR0D7qY3=#@umR(#YESINj~pC75@hjyY74a09?cH z(p>6~z5tJorFGWn&(N1cEPQf5@g{=N-SAmz!r*dD{a;A-X=kdX7f;sxB;oJG?$We^ zzMio5i*|~|Elb;s$_s65VK)^YFFI%Bnmx5mA-;{Esq~@hh>k4Mvt`n|8geEIuzE3J z<=G3|vJ<#j29&cD<989#nW@Xl>JI)1++t)=SqPwb??;?z$*BiWc1rXJYMLeZ^w_SD zi@^C-3PQ2dM67S+X(oB6`4jIFUu83#bP`|>@RlAkjnS>sMJT_pv{G1Lh&e$cm-}<*924HQm&C8=$;(T;NP7o|-rQw?N@Q!Y z$rfiD2#5*Cy*(-^iY|>(xh;z3(*pA0n%eH3VnsQOv+0_rxD~WYNs6{ zN&3IeB0DeM-ZG3uUmXec#*@~Vw)bL8+nWKvvhW1VdaCsY45 zC<^q7!lFEYuWQa+K$Hcv#LMcEf|T2g^hsU~8`M9GWj=-;d0I^-%nTZDWreSQyk@iE zJB$2B^+~0QHn#19J}tb+y~X!|z39uLLLLKkJTz()S@fwfcM^LdkwdLFr!>yk`x z25RR+ZfJU*;;F6r@On3d$EHZhmY;6=1V$a~(<_a6a zix>p8Q2(gn$6VG;(aW`sV5;8&)XAEh+fY>G}WCj(xrz&9fqG+!?p4hPJn5JleGGF zgxYE2z!iM`BE@FTF8#{LWXEe>;lwPZ1F>3DI{8*)f`UWSGTwF~+zDgrUVZ@R*7>5V0A(CN@oj~uxHqNIlY zA)V2^87TiZr=Ez8^s4joS1@l;pwwsRu3hUFCR3$-EQCqE4iXt|aBw^BQjsBM zmcvZ{pvI$975+{$0C2(1b*eAS1Qap_zq*9G7M>rHR<3NW=$gidT}z?i<1YZjmcUj}s=><^bGf8qcYp2|oTw3WQT+Y#=O8Zg3Ng zEyOXLJZUoB#~G$%L)}DqA}FX&$B9DmfbQicdj3O#t8@{tA+jLQsKAz_bkoC+L@&WR zu!#fwN-+0|?>MCCak^Dkc~#`WVGtdXeI863+rD)(=T13DppEZP+1|A-S_Ziu$uszw zYe^*C6TJN5v^`IQ-ANy`5~zKe3p0jHeC%BiwYR9FnSOj48Tb}SB_&;)*?nm^xS#_k zdG_nUf1FqIAD;(&IE4ym9%}l5$^^>V*!K)$_mlT6;5_Cz*}Icj!xL zdoV4zy&;+U^0+{zK|cdIh%xV&{_Tf+?pF{unU-^tHvjvNln6AzA_oz$nJ#F1vNiEv z7Qcw)1?K!%0?_~ul=|}X!y1E#>37339i|Jd3Sb|5z$LVK=L)hFml3X-ujDsus<8l+ z6yQ0WkK%hJZujf)0+u$&l@g_ONk|TTE8YU`VeqidAVVVm=U+Wxis0g19n72&fZnvn zvWq0T;+7U`-Ly#_LM&8080?R9PB;7)Qqr9PKLb}djO$gez&ZV}?zVhmJEROBK`dQ)}d>=%4_LRsk0@=vc@A8EOY_%c#&e2eU^S3 zJn>PC;XJSzc-vlh5uqmfWop^-`oA|{5#Or3UZx$=9JkC_%KO^hK^Sl9F0Ib{CGT+Zcmfd1sE;6SV(Ior>DM{}WW9K_^ z%7MUFrlEVdM=E?N&@75gUP*!VNOMzW;=kEPINlNlirReP`GxHnWYf{WwaE@)F`WBH z1MiE_hTde+Je-U3rf44GLr60%na! z{C$+Uc|Q%yx6-cA3s@^*udqy2gDI#ip{%L+TzaGbU_}df`L{L^$y(k|(zqmM!< zLZ7kTzneBfjOHOG2Hdcil$<<1`~oF`kP#W5+rVA{4*p(Gbx|aq3~6ERs`Tj<*ER|Y zOk1Wh)~DaAaa*+EqG~s#8wWI{_vBsn zSvPs#1zF-cm#-4+aDeD-CAj=~BYc-^`2ZjXQ;8@90>M0sTbb$b)kCr^z?Z8i4%$MCh~6 zt#1<1{C+PHTxj~e*Q*Cq_A)V325;Ugx-)@wK~eQ-{GsqR8chxvpwOMMd4=6~hw7 z-WQKPLiGI`vkkfRXqv|C6bga6ZS%P#HNTNY7_+k_zA4bCrLOcC#FPuD<@pnlEAKG6P5R_Sl|laF|luTkb3+C65fA)bZP zkRf2uEsiWHztHbi{675IXXpJ_kAxKXWm|my-t6=hnYu_8BuhUGFHS;R7yljoDoH8RHQdW0g)yIh#?{9_$Yz}r7A5- z6+%F22oO*a5fCGxmw*Bxgc2kn1QHV76P!_To@cG^ukZWbwZ=bYT-V%t&pl_KeRlcn z-#(xWuy0tiB{`Iyhy@SUS}Zuav}^$Z_l&Tq&@i8g`5;gfI5U_iJgcX#V&ntJk;U(9GFsLl~8rxfBpLN ze7t5RJmmKvv?6X&3sjbT`T9q#t*sruc;KK@$6XYvOr}m-_KQS_kWQe5j)#E!*Pc{s zda;WBR5IPe;;&BxvUNrY=w_ffd67NW-nFR3ZONE-ydsXM)ogpiTIjp8fQbUgXSa6Zn&vl8}D6VYjZO3d5!K0E4|U z^1cA5I5^HAGkdY6)ll{5oQUk+_jw-VIEuuvPXHo{}Dzm%t!b?Q1(Ci$@M}GQR@!PZpoxDb( z{nZ;>T462w&#~t@40bNd-0{G|t`CtZfLCitS^=U|!t6Mnzv39EME@+m`+#Ty?CO_q z-M8(l3t|oNECX<{D15bq&w$7G5@1JM4f#vRz(+{Pe0>3&3`-pDW6MSYS~q;Y z0M55n{|w}x8Tm>imZbH6;f&yNEI=iRcV^}74Y6z-Zt4L3-m3&I3Bv$Xi_E{h9XaH^Q_a^)j)6t5Tq zhJJmq3NZ5@W6+W|!LeOU?Ueq0mvxI=;*45_NIzk`3V4GKy8R+CB?vhExq$X$MuU`t zQ&*A<2kt%iF);N|0s$p%iqtnOJiG86k_@!amyL*vFc)f0#p4#95{bNonVxa3AM9wv z$C>)<&U#ExLP8HUIY#$JS@1V^d72A{W%L-=ss`T|T=+xbauLI5{`ONNyM}HWTy5OB zGN2H>$vMG~Xd+eH+ySVXtj>fsY1$la5*`8B*Rp>aBHEG z^48g%)=BX8*y^k)&PYWQLyAsr?ycad6dHf%?Kf>%#$P`Vk}yhUMyHBqY5hnhQ?k?;N{P z^{lyiBX5^WT(z-Hrg?1ThD~K<53IM|G2Uqr6uYQFRvKa(B`9+XxC~IlM>KL&{$$zezR%Wddcr!5e7Z%S)5vr zmkRr{fJA^(?UO`2&cTg(4W@*3vtK4HJXz0zbt6uOy)<{dep2CR_D|^$!j2|ZUfF0U z+a&E+0|Z39J5LezD&`fZ~}RX(R-^EU{gm#MkfM5FT3x9p$QrUHoW6??Ie zG5V+l)UWifn|s!+OQ<)??0xmj zwBjR|{T;RCN=YuRh}Qry_L}RH3y*W&bc}aDz9F=^VoWLD1R7;^G<-c#H9`H>;|pq3 zbLw?aRn>pTwl;E>;X4&GWfQpFQ-8jU34_8OxS?h;e!-!7V@_ zB&84cYt{F~tudO4*@UL9ovZS?OO33QLv9;L@D*5E!$@_vtm^1PHo!#sGS|hh`#HFP zKeDunPFjri={DcbhNTJ0nSqfnSGpCyyLgs;fHYFxJy;ZJ;&Lv`)z2n>C+KP5Yj`r? zfzd;DPkd6-byIsr^FE70%E|FzgK>9#y1WZEHALm3%H81#bp5%o!RmUEW7G+nlW|2x z-v^cn``wBDnDM4tTCr&MPLTSos%dsTiK81G94K=v_b#HcksX(ivMTB*jSU^N!Q;!M zIc&WptoZ^Qi(9PV>ad#Nm>P0$v(rt_6kCdjPm1cBFw}36KOV9EBFBVEVBc^p&<6F) zJGAlE+z=hM3>l7mu`BZS=8A|=UnvpBP64v1GvD?|adeeg^$=uJm&04l($04(G1dxR zH+B2_PZ0UcykXP*>|y{Nee0@R7(vc@bCH%)1R6W|8WB74d`2ID=2#orrQ5_7Dn+y* z3tOFk=sALPIA42KJ9T27LyKB9uFxF22{ImG&5emlj*QFe2MxiB0uWnsal(T}aHPJu zbR3lUA>V1E40ICRX*RCswXWD6x!Fb~e#*Xjh~U<*WfUZ@jVmk-+0`84>QERx+u!dX z?2|t+CdzAy-9`IA`!T(U-|ctzeecs$fbvNnNK7qjRfLt? zZ}AB#Ua_+#hoBVGkvQ*staj&qRH0Iej1m!n{KyDKAbaCG4xfM!-r~v61gdxAFKQEA zp^;vIv3{QMK-NL{R3#?AIqFh~sx!+<=Z9dA;q27E%|s({8eb1hM7FeS%JVwa*z#D& z1?1K})Kb*PjP{;b0EURuG>xw;$>&FK1Ceuw%#&m!6KEyc<4~kk@-1NV7YF-3u`5!E z0;U@;%k#Q|vrq-r1@Y~_h_K!rWk{3blvY(Oq;zYhYhL`?xa%&!f;wdOcly%`Sf}1- zqoB0aqiVoSf@~@`_aC+N)N!3ht!)#mM!)$k zt1RisD~sS=EvvYvEDRY9Sv)Y&8Ip#Be0?T$ z7of+KY*brTXLLHy1C1dqcEcTyl2?$%%%llk4%^+lEY%ju+u@=IPuq17Jja&ao$-BM zThjDEdxQJrkt6SHN_6UErwCYQs3J5T+C~dM0BK6WL|HpKh{~m>@YwuBVMFEq!f*87 zcq(AT2>7WLI;~*9za^!N4j&(itA5N^j%w$>8p}rK92HVG~8kN4nl!nYsJ zY_&dR4YR33-Nbo|F7dMZ8k~$P2fFk5q>FFe5R6)y?wUTFpeMc~dV0?L36@2RDW4x} zrd4W8NpmQBXwyN`rWq3vR94&(=fBNrM-EQUrFDviX4n#vu4N>8HFb=PT+Xyo&b|Zo zioX?LHR#pE;jQvBXj9&HLd|mrNqdZ;c4ShsvO6j*-<}-pj%#X3E#eywfUP{Hr3o?C z{iQc@S1Kcjr}s9cGzh9H%9XdI zVd^{0g?UfB<0E=snWTAM^vTS_THiG#=BWF(qF=AVU{E2%pTKKZPl>3c1t1WwabhNE)~suZfT){S4YVwLS?-t2<6K*&2B= zrPdv@yW?FhHa{L06qghm2Z`AFs%JJaF-I$+*oSM`m%}+CGVU(nv`oniCs^zfUC=1qNGk zI99pP79cV5VZ3#;I6coTcaU}Al}MYSPykc1EICdeq3%+i4)xwOn{5?qSgi(AQHeK3 z7FSZH65+ny^IKD1y!dtvqxV&AqvW;@xm$HB z1?C!YHlRqqd&2C|atA@k&%(0JjM^ln(F3Xdr;W6JO4lb^?rYAjVlY!}cX$4DQYK?H z``0%yTH!`cs=c`60MN}6qxSY+ZTZ;9Ry`Pr$oGtoa@tF5H*pLi=ep0&Zb=+|oEnE* zxQKevBT5`&AP_|v>@osjK<`9TdM&#Y>g6E%p5u#IxI#&aZn^Xs-#8?3y=jIx{XI95I6`?Uvd{)1yGG05yX1xGo zd1vF$Q3xn`JA!@JD1UQgG`KKpKpDw>OYyfGj}_FUJ+d>nL93LjB!3Vc;}L_KgTyTG ze+5@?)?CZlHcC(;>%BH@by>qm+{;}AxA;wExwwtc8sw3qTuz4IH=A+aQu9Q5jdu3j zd@}rgtJf}qIit6qfb~6;=NqqkPu9YrhIaMs0aPh*0BIdie1o-;z}1As_J(-T*m&Q| zlxUFU`znC5H<_QajZ}&%6WvQgTUrS7QpcIjMQ(#P25c_|`Z-|MF2pF#b~GOXrkN9E zmRWj3vTvZeNm|^K^RtJ#H-$g%J5NpiL+p3@B_VW>n{HNlyUB+B+gsTCv74TgM3A&k*1h=i`^@1`BiHOHbd_4T{R8Mdb}K>hLg32;%pL zgFP78vDaMFXI7;a$+1cW=Y4t)98NrvNqrlO#Lz{tJ75rtnH^DLn*Yz^3icYyepq}n zc!!0D%p|3Ag!6i+N&~ph$|Ip23X`RB9j_q~?jE%!*1<=vWgU$6q8S)LHwdb5FuZBC z%d&GYI}04a4^x1HIsC_t>Q^4eGdmF{Zw)uuBqyOB2J8Z3>!3_N?j)}z8jHA=n`xr>NP>bI-?7}1j2 zVE_|Ody;yY_TeE?8@^Too>njl!+;wTv*yv0-v^V9S74e769RS-xN@R#KO)YspY-9N zr@^*y1c!8vE)0DppI{_zXA0;LK1shWl#76hLk9?sos*96MquKu&iwNN?%HHtI4!gq zWUOh7`DGFk96ZL@eKI(fsfO@VaNCuBK2o5U{&fVQcnHKq)0f{93UUGf$?+x3aPCtE z`-{Q;XQtqb!7RQY&Jujtsp42NX!Q7H@P7uEzr;HJ40d1n@vq>A_yUN3esQqG?*%Eg ze+Kf|z5i!M{*jTT6wyDs=5x&S|6e@tZXRcobZUOYhWakb#a)b;T{tWn%h4 z=R}ZE19>%&h?WQfjQmUSXGdpeV6udX>cgsqLXT^(?E~tB`#6TUAA|m%!5Rl%AdE5u zh_ATUWAFFNv?JPpnw+z8{hrVw)TFM8B+K@H=fCLKr{mK)NPZI{f zA-YtM90c{W{g1K|4l=J>?CdY)+qZXy^$Tx*WUZ{eE7DqTWT_x8FR!S$b#}!Axj%0d zBT@i6Ybz*DP~d`EE^?nvmWjXU{iwLeRd8|aUg~dR^bI9zrsNb~!tDO1bjxK7n9NP4 z)34g0S{NCC;)EqV$6emvWB&@89ww9g`j<0fpQb@Vcbm`iwmrI4M_~l(bLy=n%CxnEE5^c859G4FgiLP+ihHk3yG-*nM@fc&l&>zk zh9Nu)_U4~s5OWYJjes5L3p(NPaIHyQfNLitXm+3z6}D%)1AU^VRYr*po>wKB%`&Z_4;gg|Usb229m)b7pH*)W-XgJM|-~ljTZ!a0e%vwt9|%ZU&zO zxENH<)Tp~n$@G&^Euk`y?a~c`T3&GU41XwGG*|bV^Y*AD@FDi=L>Yc} z@7}FS$O`!C@t{a0*3YQ5vK6HBXwdxaY;#oD9QgXCbJbME%Tt*-%O0&xZ=%cwlq9_7 zH417w03fr;tbcP4f_U+aCU&8Xk0Z!kYH-f=G;w2OfFUH13?nP_rLtY7`C?mIhW z2EAh6s!OZdJ9)$&)zc85xP3l4HW2T-N{qJp=1mWmK7qI6WH&JFf-sn5=hh6gyp%3+uTPkPAf$=r?v{yR!R7Ab>b?O%(F*_WXfxwe z*ga4D8jX~rW?yz%qhdwow~`oyf|&R>JAU_U2!K+p=A8|vHx$&mLaCi#@izY!DFw_N z$`oVsXN8~IE)E}DE!x@+x9iKut3(3K)+^Mh#vO#l!j#w<+l|l)D_=S|5Ulsdsx@#9 zj`u`1ja<&JiVjeuZgEu%m}&(HQdGr0R$Y}Opsx7y82Rd{3KbE*XM}Q3oz_!?VfAVR z{!UB*Km%J> zab=xyaq%n*!I&LDUVPS%Agb|wqwi@LNF|`=! zk;uxYsKudJz{Ihm=B{<<;jXoRnN#(8crhw%l^Zk{myw4B_<}%MUo*5ZU7t3VojhuX zEVq`MEbUBg&YnOHUMMK{;TxA4{FVnP@JPCzeH=9ES|v8LN9>lMYI}46-Ow+W0tCgK z$;OPK`Y7q@Y|zqydG(NLMuO4&zYV}My@Rb4IPGIM8ar(_)zxD&aVL8^>Wpk8?aNfttpV&zgN`)KIWmJu*yu#cA{#mI~ zpa8YsQ=Vh>_Pfm9vSUGMQtEhI0I?E;;UoquzB+v-Y`HROiAV4OE;3HW-AREi1H+B( zH9C@r3agMji!GiXILG+A6$?v?K+cDs-sL*}pr9)=uy^wh73G2z&Tib|&Npg4;IxzY zxDQyhpO--ij|0s4dgLrr2j)ClAwFU-3-cZOv2EdDV&acUun&-ot;Zr_PAPEWpoN{u z|9(Vdqsg& zBl(os_XLNDi`_Dr=tSQG*|>9{#mqMXd_w4`b=@Km%1n|fa2h;Ul~##DecY>d8zBr- z_cJEm(N9QyFDY~Tx5c4E=gi=5pRovaJN>3n5|KNSjbsU9v&Wc_AEe{MZJw}DuhliRwI z>7^FonwIUD7fYeVK#84lJIZ`d>=}Nshjo;w)JTRqpG`Ksuw2G*f*6H6))h_OZ;1sT zPJ%bS2e_~MwScgvg{MSV|-h{cubv;`f~)F{4I<`|#}O3-AhLjh+mm+GpWVok^O9G4%G1 z8n&yz%V5I}2Qh(oCnG!` zqaO>hPH`y~pdn;}F}jo`^Sr{I=(Eemch=`?t9C!PSh@MRofsRIy6HHIUF%%K>8yg8 zMP2feN+)hRDa;`A6~t51-d#ARlz~nV-PiGylb6pdV5<2UbnP`I>Vpc;-mEz9^7f{F-d^aR%mzrDs@yS;fi}v$?gqd#vwRzC z(^+kBQKCYj&-m?;k&{Kkq9>`qDxo|AWka|o7}jo z=}0clZ^7`X7BuX28zs>y=-ikG;O{cKw`@qo%Agbsm@ipIyN_L_@|aQ!!aJ`7wSh5u zY!1LE9%PS;fTQipqG4z4Sv zgaWZC{lI(KMj88?&){L1*d1TQpXW0Qra?dZE>pju-lhSvPx5;m^#w56%_XR!d_F4d zK|G3zxrdoKP>7kwbt*>5LtS_pAz_;*Pa&n~zV$0xj0v_@8Y42Qa1u8lg_%2Z;^hlM zQqq4NeZ@|TZbVp4%kBP`(M+$3ow>MCH|xGjI=Pm?5P0H#T*+hWHHg;tdHnsQvZlsi zM+XmufBjZ%ek{W24PXDyi$(!oGTj)B$)A zJqH?B`!=P}O#*MeO9TeB@Ca= zKl{gW1!C?w;#koRH|XZ{IHw-Qi@w>B;efs=7xt|Ykix1^0?p|)hb;kw)oZcBzg#}Xn z@yH~Mn!(D6+1$b6{@!*LR7T>s^phq5P{Yksr~xyMWO@tgkLM zCjY|%qKmTl4OSMIUu}5oCl;)ls)zR*RyPKDwF0mbP!_x=SazymTD4t5)f)i5@&VPu z-%5ZTjEiXBuxCFl=9II-rhNRrXYIi_&stuyhMBJ@%;zghqRJ>K`ato5HkIFe6G>?t z*obr^H+^M;R}kE)QRDI2nhvPD;*?Y>$7e)ZOD%-Z^KZ$v7IHcNYMCl z`R0Z+GK{=1JO|iOL7vjuh+F&J=XlGK9iJ31&<3y8wtrCc!_D8%LIy|rVTJT!!jPsOy_S1x{ zfB%=w;hoC&l%2Cr-lB8NleKnegPkt7HBX5$;k{*nsb($8h}6)}b~9LZ`h%O zroV<4Mu17B3183S>2}EoeSl;%zE#IRch#C86Cs^J*?4Lk`RYHP7OWJ$XEc1e_;D%8 z9snNjSl1(Nf|zPxtX|7)#1DF*hqI04^5eEXynv#boT7qSLmboDaq*}f=SLSEzXY74 ztV{ZH<4sw2UZvkBuGj5WjXDj3{9WGBRLuioU;u#j2enh$*?V$F4XqU>H}24Bi31tw z_@?2vyVm#KN!?X6fSK*+vHYk>|DANn8R%hbH7Z36BcT+J>|k(uD$qrW*?+JmZv00L zMQ&(4mw$Q4Mr#By46aLE~R-`&F7?0zbg@x z{M=|ggA#>7coQ2!vR~=hx;RRKfSz}ny*>8-P9WAky zJG0Tqo@U1Po=^Ae`03~ole}$eQL9-KZ*Bx>+tj7T;z7;)41eng*YEP-;Qz3vVJ-gN z^AEh8^QLm`nQFy4Nq!0mf$CFPl}Zl%x#$mSf8QM2SAl}?Z#K;T`8BtUF-B2K^@g%< zotoIQfQT9c=t`)&fk(o(a5y`=?DPVWg#*XMZFoTR0mMJ4yHCw7RxFk~oJD@?gVWL0 zFxo36a>yR&%pQlclFRq@_EmJ<)3BDg4CpYE{yTBH0e-PvS~_c)VRUnCY(1bo_QaUv zDk_+L>)Z@D3SVw>V(L`74at}ZKdOE^O!m7mC8z<@{PN7fUQGkQG4}VT8??jasO>%& zwq+1H3p*#f%xV;ew?IpZJMaojq3uh^m2j>9{19$CrSVmE=QgLLx}@1Pqsl`2ESJ}F zR;PjO!b=A!2@n=ua`S*)!C$oq{I?l75X;v;bSPxzRd#HL3j50N<6;l+>NybjSc(!a zML}}MKt=53LR%1&ii=iTv>;PBBRQ(z$1dA|8-@W9v(IjJrlk;owE?A4EvO-1;+pjj z44eTTke&>unBALMD*yc2OZ8!d_&)gO`#4QN`89}5McV4~$$*-qQD$Gk#Fc9D_aP&i z6usD1y;Yx$F9d)!fgd_#p?f_+c7cD@0Jt3>QTd2`<#wEEE)MuCgCIb{bmFCpd$cWE ztT&g&Jm$aG1#L!vyONoq+a*ktJua@)w4ZBgc%FRdQJ^GmR!1x3%iI+IUo)cu&Xb$;ZTp@kZ+8BBOcXEj~x9>-9~>lkM!ZPbi!go*$rs zwsx2IedVue)aKwu&O=X%3xmUs1W0WdfX$zs;=A<1u+02RCZ7E3AT4Ux(}_h;8(UY! zDHM*!!54VvM=^jKjyrkf8v&zCWP?PfGN__ z4{W)ggAHNa&&7EQ#B4>^zLI|&3NFmZK=yQo?~e0%O>-CLJv<-oxsoO{CJGhmjhlaE z-ZzVC2VO@mRw24RA6tZ`{7B$`c>a})FAYI=5*?D9wJHS6W{8~YECcw)Oc+rwq4}_) zjCCy>)GbwHPw&xx#Q$aVYOc-zf#1NEdk4y9#S}oxwnH#5KvO}_1kpa#U#@|U)0hwq zNSjlham8MXpk(b>h5Uo~w+vMSWffm5HCaeRF(}UX4#&D zS6eA8*rDIpXv{|2q{8G?lq|4SgE!2n^m*$}=>SBYU5{%u3SgdbWpSeK1EDi{^OyKZ z>od5uAz`-p2>RQMH4aIvd{Uk}m4~{Dv2{e@17kXTGmL|a57URf|^-|;Q?7$$@6Sa)E zmTG+!0>YRQZJ!gBX6Oa+l)>4goYc+5mDfxkraRQ-@)f+SO^@Yb-#aW{KiS8Hd7eD?`qhWW4-qw**G9j$Db4n4R@;Ot8b;x!FCEKh2o8HTpf=F7)s zukH;Sh)2SFx9Z9+J|sTjUwyuOL{QSyl>&@U+x7wnMY-VF0j591^v;YhWmRHi4$gMb z*>WG*S-~j?onec3Y297Z*L|Xx5ExUHtJG|q)1+vpc3~E%L&pb|Nvy;s$qyFltt$6) zb3EoXDRn*t@%1QrKcg&5IXXC}uQti3sH?$IIIQPzcM;>c?vLIvN&B;ZbK7K~#x5@E za1NlUn#~O=O6s$ofYw6^2jR`^JE6|TUX>(Z)$b^aCSnB-YG7}+R@fQy>RUEgwL}<( zo$kjtab@a#=m+RoJWm42GtBjBt^2P}9NmwCiia{ca-1lnR4HWpQ9-e*Rb3mr@4Bv* zW$|J3b<2pq`KbBv#d}E!5ju&z0ofWqm89|P)E=7N>2=R8&&|V9d6P-%GEIis-s6+( z+B*9tU~o}+Cv*CMc>I`!Z1(UZB-h)ubZUCuS?E*FIMg7tSh#KN99KYlinj#ZSj+rj zT`h15|9fu4i?eNz{zA}nu9WqrX9GeLc>R^B=b64GUHlL1rv)XKy6s{CE4%4ruUh9l zG_LoqI~~6B`?K~o1RUSk(`xVJ#xm=E(#QTWk4Y7{2i>#2QE<5uhs?&Z>?kGQm~vvB zvo5K%Im6WTaF$teic*3B>BfZwT_hftWPKsjT9Ffk9Amt-LK3Zf)#7ibQ_64{AaSu? z{eA{}y#!ieJ_cRbBiz=D2Je-*)a6!wiaDE zHP{g5D9@iKlaqF{J2{{Mq^?;?7=4GZG@R`mNXFBz?5=4BJ!&~mXtm5PRrQi zmENpVg`OS6w!{7LeQm=@Mu3K+nM)aG5m*H5peReQ(L>jsG$I@^J4`23`}*hOK;2UW{w@4dhdWsr~Kn3_CdMiZj&1sNW?uciz2h zkDu4A99(a^to(@*Q&h9RM}ePn-z_6NHeM3rW6&8jCVA{Lz5qnt!Aa#|DJ(o%Et`$P z7O`STtmfJ*ti4^lRzZT&@wf8k^E-OW_@0EahcA_Tak19d5PsZZq5MKm@`#RXb+IXO$gPy8ykigaf-6xwxv4jg>Y# z^+VjiL4fh&a)RmM!9lIDzoHmwk!43@eZP?g4Vo!L~Y98+Jm&Z#9ggcp0|L< z747U?X~ng~Jhj39#N6E>3&Jtc+vUjGmU0K?}zQzVDwe($q*%3bkV{Dq3C`$D5J5dJKsFH#U6H-*U6wO<6~ybNpC=9 zX#8#Y^7)y3E7+(W2~muzJ3w$}xDQ0Gu9mI8rGKm8QHOQnRMXzcr-%!-*hjv(xVH)x zTHlEJ1JIv>M7X(f7eRX@q*^U1jo*i{ux{Iq#^1PBmIrpV(eHFA0B%VuecWZ;E$I8@ zPWM4>j9=*hCRi|{HHAEeu2zB7LK0H~efn}M;}(9+SOlfbyQ#b8vs-B|+i7>bIajDJ zzfoCW9w%cI9u^6^{R||4+CA5T(~L22qx&;P94v&I5iO6a9>;c_@P-gFCPFJLHNZNR z=Q<)LjFHXb3Ka#-g=hRZjuM%U11a=%FGnq_vpugf1)a^mU1ECiV`LNR(*uhmDr@}a zuY$t!cIn$+ho9bD23tGEH*ET&X1Lu^G#iwLx0uJ?fd_6snN{Lf>8&}qo9QykT1me_ z-4b9PqZ=FBjkn7o&!Ah56HTQ8Jt88`M3>D?*bkx zmDUlF#7N3pNQp>H?>pff=ADL)IleGAcw?(<5+JQx7B_$Rp$#5W&^+(8o#v&u%O#${ zi-;M?$itfkfKnTWP&^C2_4+_M_Ki9xSqN{Q z2K0B|1m49dynn9!GY9|3!KZZg|GT{aQtA9mt8XJFr(vg+h2LkvzQb7?C#cKbO3wtr zjoI~gq7-3(hpwb^jl>4(K;x{-yBE{b{tKlbsN&;7j1{S&X+Ny`0;_w{-q0Y%8bLtM zs|>Wy-K)P!bg2jl0J2Q0d3@hT)RR@yDRNn}_ROCUmo)xkYrrVRT@q0=;_p= z(wRZIHfa)1?Lrz7q|<#DwNyp#%L;5w3bq2$!1qBrmfehlg&7<=)blX_acyS(?C|i* zg8dSGerh5{SP<(0#E^~2|r_64%)xQRPh6TH<9Q%74m`ibg^^2M%lW8 z8)yFt9)YQt7#`laxUI{X?}kNvY;19sczS1Mlg8coi;KUJ;S~)4wYllezS{s~C=MS1 zZy{2D;(@vmdc|RN=9K$DO&LXga&>LuXSCsyuG+iv0VSIxX4qHGRm)ATUX}Fa-+Gx$ zPm#mZT}!u19~^0Fao+dz%cCo}`r=CxA0Ty24=)hurzQ9=~)0}^I7_u+pb?m62c_jd}wb zBcIvuWf+!EoA)mb!>?rNFMpi;GS_C$wd*lDJ!>SYTgW&L{et@Drays2Am(c5jd+uA zAB5{_6=P>8COQ}A0Q(9+2mm_a;TAD0*ddHYzMf?QY42T4gh^jVk`lqQkDq%lRFIa!paS1nkkoFuK9vY!%k*L%H zh|*G_en}il72!$Vv~!z^0b_i&;E!~&#CzS*Pt)EBFu5}$oSB&tbSxA2d2BNN=8S2` zAYl6gByZwx@_Q_#$c=UKf4VY9HV;#oiIezvlRD5bcyv$-_ua}Ji9TFJsUD%WaaM`! zB?JtCoic};h(+520IsZxDkHAoBya?JE?E2Qn$flby%9)gjOf@wAQlQXA?CKC8rj8b zKOKWIKrUzAC>z~nv)~ajwMG8kP49=7JM#4WT==)J!NKl&5IZM`DpMHa8m_iCK4uW<$$d>ldnad3F|jgyZM?N@pA+yzX2jU zv(p5=YiPY0Xjap;I9@Fl#jsxDxebz!L7a2Vy0zW6O#Ocjp6w{U!X#|}(Kh3UjC=RT zb=rHc7?ihmWXDTLtkf_z{OxpASTZWr561Ihgg8qrvh0~2#QgDrR`ri}Ch`8eetU)& zH6=(~O7@|wt9}1U_U-#LCZk!TabWXW#qVF+#=9l%W=cw|)4H3!)#2bOrsS0-I~);L zv|B>1vFv)xJ*))yQdT?q5~8afj`Da MEe!L2Kl8``0B&ZmP5=M^ literal 0 HcmV?d00001 diff --git a/examples/hello-world/step-by-step/cifar10/sag/mpi_gather.png b/examples/hello-world/step-by-step/cifar10/sag/mpi_gather.png new file mode 100644 index 0000000000000000000000000000000000000000..0c01ee9a6d232be2f554b12bb98c11289bb1d18c GIT binary patch literal 7653 zcmaKRby!sYvp?M_0t*O82qLwJl$12mh=k-4!b-zZOUcp*f|L?VOC!B>ECSLEB1=h! zq%66_-OuNHpL_qfzvuVIInQ}c%*=ac&TD2~?-)HDHA*sOG8`Nn%9rZOuW)d10odyd z5+dx`6K4De2Zy!drLw|nKg<0rQuq6_>3uQ$YL8O=N;O_dl_d@8H*XQc(d+Zq1l>BX#-vTlg*JG-%bo`;n%O= zGiL#oCuR-KdBTA!kZ+ljv&msO;P%@cS*}z?v)>{&2_v8h^Xh-Z;yzd@bf3v z5fl~{qQls)k%!hP?8Cge*?KA}5q=l%FS^+$iP-~lHwWI1;CWmQ8h3EsKB+KoPU^Ut zshWi06F+-2!eteEoo>Avy-hm2+8g^0RsYWOGiWe{@0mXutOT1#N=#JfyxDh~8B=|G zO2REfMw4Eg6P!_7Tf1J7+ds?s-abu|(;p9VxFmbkD;%tlz$W{5DB$8KucZAx149Nw znzZl1_-x0S*lC^q8`*IIGOUPlu<2#6_++5$@IPd8aNZDe3E1uQ>m@v!cE_ z&$s#tBOGTm;0A|nb8~ZO%<`S(gY$HU?02uizI%wfptzk+7a(Jo8zON^G1YY6y5%q1|m|S_V_>*$A#~6V(wc)`xf2<7b?Tf&b8|&+On6v$z zj}l9IL;ZcdTRFGrVz2dn+KFLU&cMhx)!;I(JYqdtY2lb{ z{*xka@1)%7Mx)(`(45YZmF%y_v6P;R@=)#DA@Ie~O7MJAa`L;K$--TSJ^Vx)03)ph z47U>-`kjSl$m(6OPF8---Sw7Zwg3=?E)))m3BF!qH;82r+&)_Ac9bftFvQ#E$`RE| zZ)th1ANE4f{^)lXIyvV$T<*Mitn~7)+=aQN<&s?{!?f0Dw(OrqL3#Pkmm}(ORNa#B zZSrBb6r?2J{9yGiOWL=n8M0LdMS1>J0*9y8lG&dn4#8!hNgb!(+(`AY7RGKTGzVVR zu6PgS;JaY1W;-j%GQ=b1$9jMwJXd&OjX<^%vqskmiOx&yvWf}{lhHx^w}lmwT<#Wz+>uGZm&r ze3@@_!?-Bj#hhJ(2lG1BImQtK^X_qZN>NuX^EI}_?Pn;bT?Z(8qz7&dZCZldmAbQZ z&{h5N8JLWMgqs`Iz(?^w^3!wgU2aD~z{;DBK&hbPSSz&j-Fe5k?BK{qq2?E%m@!zT z_EKcL&GaUaouoiH;A}51=Vs3#7{UDCf&n4KhCC=d{FEY^c$nR^*Z1qpL@u?-gkUt6 zNX7Lae^qpPFX}|tv_87yBkyCU78@m4gw|g@B64-|_ z0hG?{whvr$`bj7r2+-fZPZG7@{#rKR%vnQ2VVGkV-+t1nBPe!G+U5B+C zr-~@s?`|=#4GkZ$qeOpGhPpqZF}%F}_7w|l;xYNZ34h7`@avKLNw|4&yf$59+s7jj zN`iZ`HC9Y5f34@|phUra-Rt12*3m{?20fZS6k}#)R(6-X3_do3-7V7b>O_M-X<(Qp zgjU4qD#dASB*y2-#N?%(6q#9L0Bal=6FPuoelTXg@kS_c?^g|Hk#6-U&w6*Cz@XE4 z-!Y97ia*yVQgR*@piJj^ey~{73A#q=wWq#+9DN<#q~#j9ZyVRQXKM+XoSeKCYVbW= z(uVtF5+XiZw)W)!YW+EA-?S}!K7{#VH^#w|}@XY|JmzWP53Yb*CXlF<3zU!WkBtxwNBBoICj~=NA zZIb=tn{l}=LL%#DD1=k7<-&BSfwu8XC~WP8eyZ)7jOPMJ9UrtF4yrt$E^7ckeR3W${wR^i zjdNJ=)hMj#mwv@=Bq25dsOS4kK%+VGRhBf*ZRKy!2F>cqP5!54R`dthQsGIf76woq zL=+U?-QHk46Qe_@UhI>w**tB={YQ60LJd+Y4XDr#pm;wwfX%lVh&EhOf7D07PZYfM zB)4%{2e_jlHT#J}Y2CSR#S8QM>WGK>^IitiA1@5f9D_d}4i-*yOx~(9zaJVLT<4@M ztgnA9%l^D+;VCJ#CU=T31tf@bM!v8wjF`H_Wxiooiq_>-0$fxoB|VerS?cF6gRQC* zyz|7vnT@HY_5CCnJldwhhs?BaSN7qmca6?Wi2i<{R?Nwkfs4zxw$={h-eRk9JIS08tZPN(+m+ZP{icqQ~#YAiS-eM ziL=-N4ts=`&L1sZMtPUCj)3DPE+=ZGH>pKNf4W3At9u*^acIC(c(2~IiP73`sBy6^M)5?G?pK(B~Q ziP%&Y;u;1(ZVZ^FgmEyFQpAa!MiYj+ zf+Fr-a1wXUpEV*^h6-%Ojo(AE}gUoD?VBa|qB)A6f^tq{N_ zp3jr@CK^4)D_}(_Tcxw<(4@KVL5iEgf9PiC{fr*5!7kt@9$kFU{j#!hLBxN=P6#nY z-I3PCWdjXar4-ewQX5XNNIeb7_mu_c&;Yc^URPfJ?h5nt(Nm-a2!tAp#_TNnW5cb8 z7)yubL?4#CoSa&HRs8#2v*q$q%mz9BocJ2KopdIGQjQnRc+MwSUgXZXs zp)W&oEftnIPa^3xHHN_cljfQdR1VJvrDiMMa5+eHk>Z7=mf3e!{1XQh`vzN>lw6GT zy`54uU8bTZ8_#=a06-Fy9R!LQFXs7D46 z_t^0D>p>N~&^24uNYBx>HkoF1IwNWpRbtD)3E+Of&Z9x|=2uH%HXuZ^=LqX_uj*+l zdNRO-0czi*)BC*gmfC3wrd^M=(hMv5p09|7l@&XTByPgIniXhVp?p6|%5y#1oI4+s zl;|lmObO%yyO-dH5o+dnWxugThT!fZ=JzGZ#iTdP3sgTQ)v=QPi`UqXL(XhK1pM7I zMo5npGDRsyMd2n=mrRC}s6A7)h*MM4gzSZk4?eb2kij0Xu6Zk#_k>h!`#5Fa;P`ZV zocoX;u68hNLs(SGoJQ|X|+RM}uR|Z#?{sDWQHv`h#R$xy0Cr9&C51-1pgY?|nRz zwV9{@GdMFv9S}2hoNUcZFgh7u$uYO6(@5Tsz^1#sx_|S`G4j3SL+W3=O-P*K>nT;- z=x`;Qiw85KHhW8T{6W>7@^3Bt6PVe=m<~J%FGP?IHD~{Ht5319=~vN)iP!DDWn1wp z&_f^y8*~jc$|@%}ZSTIy<_s6?Vm2)65~N_b*arQC#qgEuQW=YO#rkvuVIt5md`%|s zwXRx~bMC1ck%9Fk6V%u^BvJ5NC*{|({dogNN5|Y4bzB8r_N6mYQkUy^d`5=6$ZB^o zF~m&Db(cJ+4g0q>?#4M{;R&TL#+Si=!nUVs$9QU0|DLbmVA2=?I8{E~QPk(H;QFoH z7WdO)iK6)Zel-GI7il`uEbZFHw#i+m`b)opvSl|CVHphpBnzGk_|lU0DPvcaUO<== z*)`30-bcgrv~&DU>9xLu^YS@9@yPP-#~n~-uER524S89y%6&-p`QIMR`@&OE2^pgS z$K!{68AzPRzb6YdGnM^b-@A@o<s3?T;omHo<7~LztUrGaUb&8 z>S+j}a=t($9xiI^N&#Vi{}xPgI7naU9D^3iB!%VFzRMkjf_ts>1xm|1c`Y+C6B3Fh z%bYxY=wv|bfA9)In4*vTdCQ%v!~aH$JN#@OZEumTUqjzVLG2@}J~}Q5<{BW)@J}f9 z3`CvyQ@Txo6(fT(5}Bh6x0-Sl{LSWYCH8Ez?+b>?paqxl7;-i4(v2Zd!(!1pHly_I2li!YmbmgdjpB6l zm8oou+F^dU!UX7=ih&(XV~srv`!x_JKX3xC{ooH3^?j?{`Q<(PCWx@rLRD%RqRx-g zlFV3Rfi+-3pz$EPm{Kh=nvap1t;Na6Kis1j8FCecut}q~;_lF^s=Jxyp3w5*O7pDd z^OaNpw+-J!I-=fN*MP|w42jN(Xo#DPLHCbTR>^xAF}YoB5YI+^li<(dY&U8Gk<8sh zK{S`X3w-hRu!}0$1uHadOJfm}$AT~rXXNYJtaz;Ouc4VE!M?txgWDs#7lKpYSWmi2 zTC5cBo=YeDz%vw6z4Mye(7+`cK*Zk+mip5klV!$iWijf(4d>ag_{n-(t^sXrN?|WO zV#W002-=PxMdbk<$&XhBVYpz$-^AaBrQB_k6?1s&2Y7+1O6@xNUw1!L8A%}_+L5aP zgo@h}0)72F1+m!K2-ncKO?Kk8&zzu2PY=%J_|6q4nSkT3CGMXn;L~tg*~aL=xZ_IX zV-VGM6#OgdpB>t-l=2>d!T6TWOIUx~9 z2yaOSY?Xep7!>APCXUcA%ot@3u~`HWo+T_=n$x`s9wI)Z}+Yn zV+xU@Tu{?5ymZf-_*&Z#ZDfD`^(yKk?!|(-cu_4Z^9hW-^rP+y+* z;w?0jC0Zj}#UkqW*Zl@%zU-BE5%?&4Pvg8LQ0>~6a{!(@cLD7QKr0lTKzp$NSsCt7LU0s1a7_V&0X?&H9f4QDlbr2S%laqKKz^mVj`oX zT{Q#UmphReHzRY;KA1W?9=P|Ny=w+Aq>Z^3 zkEPS)xE3Y|34NC63+L*{Aa?{F}k&@@Sbnhp`2U~m2}_-$M8 zLeGpC>(`9a!F2PO@%Q{@YjhXc^iy= zkg{P#t>CT(1YD%eQupF80LoL0t$ZuSnA%Dw*y*%^AKA)_vXoD6o*v&g%Zlg(T4@b~ zV5Ee_S8#oY3Dtp#XO7<0xb`a2xLSf>xa6E!puEndB)9pE!S^a1YBA(d)5bJ65T0gY^ zJ02Y$ux0&zj;LR(ZB|m&Lk%29*BXNIm6|6V9ygx?ER_7Pn$*__azx+G%&N$TTX6Ai z&m*hdDi$nEC^^S^C>Q|J?xAB$7aw3_0BIypR;Epb{chCTQhwdf1hmKfS8Yky|MS?L25Q`m zM<-~r0wMZSJgyaMXMmSHE5x|fpXst<)C;YliVN#+Wnf6#z891u%AO5>Xv3$1w*hO! zw4^@#yR9C73Y4T{5CGQT&~@c+K!~(6rWdSf(`Pb*{mV$wZFn_kpu>EgEtwQNkFhUt zn_^QQ&f;1uu<#PaM8q(n8W-pyEs{ORx{8-V%EFLlvbF^&`p%Xky8VFC*MupScpq?P z^po0y2CDlVZ|(jbS}JUl7H-P;IVZJF3+63Hl6Y^&)ad05y@s^-hp_$tMg{=yar#&~ zE+>O0x57RMQ~GlCZZ=4}?sr{W&Nw2;cJ+N)ALMzcCUwfGlF3ky!n!Oac^q9$_ZCG4 zC!ki&J18nGIc$SKHJ2eG#^1rsE_(biRtG5-(bpdSJ;o=GGnhD$3d7P)Ffm_(?cWhf z2h#=J?=E3f_2`0z*Lr~_LYsYy87v*OtncR3_TSyCD5Rt|hK??y_!&jl8>;4fztww0 zh!Q6zXqtfFIt63tJ7r_(EE^6zb`tyrQYCzL@NA)Qm6yC>7jX)7+GcD)k7gJg4XBwN zdy+!A2%V&&1Fx};V8G^)S^P%^>cF3c^q#az8E#nwOs{T?RRh%x7d4xQpa532ysb@gwS5_+jGX0@w#c}QgrE28>HpEi9MGT_<3 z{swv&>uFLEla4TY5#Cowj4H)LoS-R;3G8e+smSBZLUPh6)rh7(BuJN0yK4CO4tbR7 zCwKjY#Z67E3&@{kS*e$*-6@RRY!D5Ubax)5<=Ngt9O;lTVmlzc5OqUt*i#_7u znT)380x!0Br{v_Y(?%I;Ep*}0xlPZ`PW}95R*EGAgG28LcpolhO-@bitaPiZ@quJ0 z$hF!+nESD$d5ObVwsudq#_oKhFsqz%n_iOjEjvS+Uv6F=Y;z=YR#`xg|4oB4Qz^3ovA+oW#If*4s@PGOkRhV!Lip1dx04( z2(6nVmar*cq8K6JHWe+B48GVV#gZCl2aDy|kRNeOo<1)orM)>Fe+~ob#oTn<-27rQ z2rp;bN~GFP&xoMsJf+YRPJgFvo@F+>v-a{>Orf6eG5?e9jXtg#unJL$&zviOf&Gl%C`z z_<$dX`i~(0W9Y{iI^HsBOjj83hO>oHm2TlzDe}^<^>IbKBZa=nTSF>dU7Q*la1%KY z5kp#22;mPWP+{Fn6ze*pd0zbDRWuX`&@U?~Nr~Ogr$=BJ>i#)g>;Ek4$4I8cOjNs4 z6jeU{Po1*z@|j@vCikX*MvoF{fK%Vc0<>HZg@I|6 ziHW5;Iyr5uVh2tUM^@<4?zlpWIaYD-7L?Sn zBkJ~2dqB#UFTA1OKC;U%xdqYPi^n%kH-}h(L%vv$z|LlBn0B7Km9;$ B*mwW{ literal 0 HcmV?d00001 diff --git a/examples/hello-world/step-by-step/cifar10/sag/mpi_scatter.png b/examples/hello-world/step-by-step/cifar10/sag/mpi_scatter.png new file mode 100644 index 0000000000000000000000000000000000000000..421507a339c645b013b78eca3b9d04e26934809d GIT binary patch literal 16160 zcmc(`WmH|w+NO)UI|Nu*kl?{>;S$_EXb29$-Q7b%fZ%Q+um}*`HMqNz6z`ESPQHqpeA;=<$nC=!Lv9>%(8o|koWL0)ea0Ksqg7aj z_4*eLT3PVxLJ5I_yb+#3Wgc7=o*{+_32A8oEktY;M6q!<9{xMM_35>P@6$RBPCBa< zB1gCfY|SS^9w)$)b-yvN--+7`Zmgc(+~S5`@???bTX1o>JzS}sTRmLyFCo`lSeEE) zV5}Z}g#Wk4W(n7#r=5s(?|u>7KtN^kBJ2&To|&dMxAPsUUMnuB51Y6_c{hJ|?&{-^ zZtj`Iue)$}C*?uKMMVrvKFZ{Lb}9bR<}2ZRTVb3Rpr>p3SDc(_-;k`VtlY=NuIljc zZ?23QoHTTFh>sR(!UAunbEjH;{v!P3iM?Q7zhr4^^u-$gyBvV$j068TgLcv zVr{>!B+T8LE}f171v2ZQ6A%*G?oQ;hw+Ek{81g%n#R>{SMJK(@*ok0Ac zep>p(qf|A!;O?Zu^nq#`0@S9jJS^?w|u@0dP-lavz0C` zF1CBf;IrRde>__66!zGc0eW}&&8Wr#x$U$Y zbIbZBj_Ik&YO3hX`GhoXOzy?f9&HCsH(J+1y?q+wj)93OZrj$@_WX11efXcTY}S%! zbaeE6u`qI;_Y*(^jFy@`zNM2KuXp5{_#N^2KHlui^5apX;RSz4pcXT13!+&Ex{hVv zhiW=B018^EK2DaW&Ocgih|7g?=G}`pfM33RS?9DW%-y{3!%C^sQ!ZJaQZdB*?xd_n;58H!&?%~^l``e}Yc3RYat7Tf z(PsJb^7Cuj*;SA|oF6Tx9k*SX&dvp->sf#dU$wr zdWHnzMoxOZ_N}(JvBA4Sr}jJK7WO{Po7rMDYtcx3Abtm%bbY+{F|VDfDnYg%=Hlq` zg~98ihklnQhi4f)`n?`zUY8xec3_r!bGQD)5lGE#5x$P5f>KEBb74PMY0$hgK%{1r z#PTwcJ6@?K>e`GCec&|@&qLzyfByGg*#CxG3<$H&A1qcHoO1 zcziNXGHlmc{EmFpWBzToe!{%{31*G z3(N*KEV)!G1fz$`rSHY_8gyg^GbigO(Dx^ctU)3HUStdDi!vuu`rkdCL@j@ZZyd*# zcx^h?G|GkFz14F)Ju@R?U|>-9VN+^*Gj3g)bsahgjF z&YV+`8$nH~tWtx+y<8~fFxYDa!z9Y3A0KH78D4YJk@692+IOMtoo@D6Ob>9pZLafX z|B2nbW{*XPKmw}PG%$E`(sq@T`{8_dQrZFAaV8OQ`|nO}5J#_~AN3*sHwT_xE=e;O zjE)R8*v%94=-(Xp(7ZB(2e~rqEfiY^iuFP2eQp;YR|UC%Q`Rej>h@c^IBXnx;6sft z3B0koSW!S8{-+EIJc=VUtL2}LuAj^FTRgw7-_&Ocy^cpA0da=xrOtuF+WuI=U(#q* zO}2PBdn&&}0E&I7``xfOw)=&1tK@vu+s~Ql2)ez%s#IixsdR%y54=EZb}*UfJM!Z% zlI6qg#eO{`iM&Sva_x;AF2!sC7uBt=7QxkLKVOyYtKx$?FJb1>UQzMtT~`zNKo9WG zpFVxkrcD?2s4%8VQCv@J^B8bCFYN%!W;b7zgm%+4y-^20URz#d{QC9F)dQ2{@<{3K z@tkS($HAn$ZPZB{S7KD(04vrDer4j&x;ow_VP;G4R~f>Rk-XM>DGEUcB5X2YDJO!I zY!R=D`QFs2e1*MP%sa}x)W~J$QYuW+Ax^6-)VL(QiAbrG)YPF_$*!`})&kA@tTa;z z&iXxudNfrhrz+pkhBaTMDku23q*7oi5V-g|&W2bk@WL%me73%l?k%O|jAJJxoTmke za@M2aI0z2$nOkEZ|NCvc7qUn+Qk$m}g+YZSC4clef8Y9?CNmY0bcLAfH*v&|=L2OD zv4KtC&52_1!U=n3E8bdjZ^2(&V4oK`GrS|JRA!m)t>Xk<$q5B|+gYxT9~v5p0CZMb zK3-R^(ahaxy1ZqEPH+ht@J<%%-}Pb?$|Yz1uJ7L(KDSNwYs22o?l)rz9Fv?wO`-9@ zM832uVI=T95u8r%%~$8X!T>fhb0JC!_~Gj?R9xNa5C~2rI$T`R{NTvQxsr$7dns;j zO?$$au6+FABP>EVAgX^jEWXyEQHQ9g*3r|%d)IY`SulLchOq&SDXL?Py2bvx?q>gc zFj?E!m`#9B#>6Ct>y`aDsp=9=|7T!CVh#kVCvs}tZ1+MM_N0C#lh6|%4uhT7M4!(l zrW7APeIm#Zaw{?2MH%v=7nz*fCfpnHSZj$x#{fcV@SC9O8`Smux-2XA?LT@=ncRF+ za;=4(jJzKwr$~j>9rm(f5OKsh&$$0yC+<{Q&rM;=Qq7r z(3a6+Mn+5mYH)-4QUwIM60ocJrNvf^56y=1SdU$QSC`dX3wjC=4ti`|_hH@Vn9MlO zue-SxtmrV@)En@oCIJb2*7E(BdU!%FM@yEl8}g;%;qEGqjLV0DxUw_N8d!HMPA<)O zSDW83o&B6BcxYq(W)^i4UGqJid>;5j<@T#}V<%DMsK^4s5qr1PFNK#{ys8&m+$ZuR zXTIri)^|d%M8>gt=fHh!L=6BH^6Sjo1B%r@!~B*w1ymm zMCB)(%i>QILCY#!AE?k`CMT1m!kpjzyyz{hyBxgJHpu1F<5>&t8`v{D57t1Ep)t%H zYWnn0N6VuDt{hI((%SW2W4wXTnte7*xh>HTvA1ic)#Evi>>K!HHUl{xM6H7@kX5u< z_{+yibZ)e6P55c6`zTgTAz#5+x-y4(a#2*zPK;IV)tya9^w4wk6GN^A@#6`&3_Hu< z3sHFn*_BE!=h_%wR4EkRCJMU=4UV z)c3WIyMtw=&UjcWg$Bbr0vS~?IY=sE?HiEwYAOk?1=CB$mT}vjo*nur8~7H*;D^&oyb-6yRfLfpy=zw;6A3cbaFHb7KI3xa|^+8uI?T8 zO^zk^k?Wk(_cRDP0fjV`cp2i^@026^m~v8Pp|M;jWWKPvp9n~K9Lh(u{?QibB_E?_ zyJHPSn|upRuI0J}(7;m9gAjx2jgd(O6n*z&XL=K}X0`O~=DSCJZmJ)lN>Fk-B4jBd zr*>LK4#;&cMNy3Yo*xFiD*7IMCDNN#l0!dwCzvMQTe7h2`cS5@Nvf?)yvt=WMg_1A`ZKfXn-z{tE;{2OLxT=d>1HxXRMx!)7pUOi`6tiHVYh z>v0(jIaAEb3kVyvV%y`F*G)qNDH5~QD^z5dX0R9%vSf3mP6FYa3Qr`{y46zap{Q*P z8FE%f&jL-PigwQXx#MJ(DWE;X`DYqq2ifv|2?g>5-4omQ_{rzC@vF*d8G_WpJaCVz*sR@NXEzS$Mj+tdWBM0@TCTu2c<-eYkM`iVTWArZz zaT@-JyTW)8NwxTuFX1&uAAiLfTb*AFv9{H(S>@l5Ya}Ubq>BX4N*rJc7Ya6Z^=+m( zF-su&Z(TJzcc8JBbc(r%Ix1m(@+fKo3_1f*SbOAI3aYjl3fk|jxiaE)82j`?e%pI9 zn+wNf3#XTv!8FtnBw_9?#>p&tBGE<`712!f*&))5n(=~@shY;w>~g82ArD+zELmo3ywdj9Y)1U=v;GiN5ZPEJ2^s2Npeh7DQ{5=^7c~} ze)h8*!NQ1+b?BMYI0!!h*AAMxame6>7!prU@*BlzoccL6pCrJ0x$@C2ovF z;Dud{g%@4WH|-NPt|O^f0;n}pZyCzQ&q!GBhpE#ajC%wfR<&QVCf!Y2 zg(?aWrD!>f3PBrkEs`{@inuYH<(d!+_lroLtD;c7oBk2*z!N4huY{Em9U|Rql5d~! z=#J)QM>i>s>r@m4wft3~K~+vYm;=J1vKdd&$2ElETf4;<`=oqAmPIFRB0Oli@YDS& z1N{{zwn3LtO)&@R2s$+fZ=4L81uy%)8o2LvW0V)Oy5wA&ExMLFhu1O&M5}%H6I~kh zR!hG3QyaKOM^1F!Iq>4Q`)kNN3j)$Se6_mqXBAeQHl`8e59-E^)lsCbs6xPKxEp)@ ze&EbaJn|1JLmEI0l^X)i34LdE%W{6#FSaqGbrd<_1!5eT7$+)Mk9v3fp!t%CpwY-q zgRNRI``FyKaK8p&FWr9xDHUbCx?ZV&^ZoQnG^arpd}w!>^$<@|(!O_fBiH_cbej_P z>(zoPPk>s4P>V?P%SCo3a53TzNgSMpKrb@)8j4UTj;X$+NUcBpQjfIlaZV!dut2j{ z&BK7uR_)(&cg0}?)#CalO3t3foc%~CVOJM+M{&Nz?>UoW&ndGp^1n4XOH!w`);AXy z7urTftd9dHBYGd6cDFkpV-l`A;rs!)!Lar{b^=H|M=p`_4ZU1E-qZF~z5TqVs%l~> zcAtuhO3IfnvY4bCMUM}6S_gCA%XPl#+5rhVHZ~boa4N@J#l^+NoCDTlP=N7-_@7u@ zBr8CPm}&F#>PSS>UGv_?&JgqCf7`f`&29_RHf;6gZhO2vV84~Fc0XGBlFDhSs#n$M z=JPh>P98*hktDRGWoW4QnNg)6%`~v7FOy&MwD}j1G#D6s^|zZ3W*Yx9npy6)GseEw z@tk9;QHgjGJq>#(c176$S?Jjw74B6MnBy>aw{Hi?6X4g`gl&gXSn2}qJcmX`ina%m z;+|k4iKzQBSxjPWLj8Rz#%;b|7?sl}wjMkW=2YT|lBmTL6A~V23B0?h!2anaYPqU! zo84_;OV2u4!i!`5Q0U|2K=A`%)3rp5UX}5pMIDdtxebu{rlfz~54ru!q^4a)MCC9q^ADn*J9n?*Ec)4ak!b6qJWe_j3UR>^H7)+ zcgN%kGpMskUVq_p@V#NTH-^xaN7_t#rc@2o@z>ns!2PuMlepmDo$=Cxd3vS$h#TP? z{iVOyAn?jQlYZ~>#GXCIDEoUJuk2l+Kw=}zO?Wph1d}dXTf9zEGlft@zw{@(C~Z&E zGY!yD78<2_`LfTsSIBXRw<+dm6X+M>17Z91?>~QT*$)!PdHQ*C5wcLH`JpTN9If84 zf|U{cjj)1N+!V*YPK~A=f$UgsCqw_X z{T&n8=OZ8>_|5newa4n{-WL$sTnV*0F$!H1*4q3sFY%)JR>W_;i-NidbK~Q&F5Yuv zu`x4$%uJUG3~U$daQ$vMkoZf*VYEW$+l*fn=XAY&%`c(wayvj(ov}A(nL~f>bn+v* zn!g#9o)eLhe~pYiS(wHMi&<4wfZbOb zCO%N&in21GHbozwrbe8qr~_R+5laD##viOf%Ga;^e3|2DeOo+_X6#8wNQPAI^+g@X zim!7@e67AzzoUv#p$i1!UUWvhzt4I&m3<$(`o{@=Zcs8)e{ZPSu*|E8JH&KK+ z0M*l*FIVA?>CPCZZ8iw9n~PsKpebpdyu!e2vYQJC8T;bb0ltSPPx7mNSB0N6w+xWSN)v2arC7(xk1QEUmtOU~_>zg#SMbQq0Zzw#B@hj#Sr~uw zjzA^-wKVhJPV+*WPWA1=65wMlpFdm{aJOEP^slQrjP~IfdIDb{e7^{2qQeI@iZjoL z*OpQ<271%cO+8K`!26#6vWE6Z!VI5?h|>h~gW}_o_yfOHja~n7C%Z_|(~iykdxp7N z%%A-%q#QK3svwO-blbr~YSe4A^22#d^=_F7V3U`Ghe*EZ*PG~>Hn;MaC1#Go9%;C! zeM%2nCMJ^f;*|2zV1<-#p%xlXWTd$YBG4FXlOlW5<}d+X{wdQ(^Ocg21LpbODL>{) z6422@K?fZz0W4%v9guCJ{weka&n@amR!i{ZAn>XPZ26aTu@I^&nx2$8DuC z6;a&nK;w1ivVN1~N~H$1@455L%nT)5l$oY`yaryJMV(gg>=t6(4h;E#%usiKzAp)imgyy#J`;8SedQ9!Vy#|Kq{U(I9Hzp~kiVxprv%C%%tjU~=z)r(JRCT^V&*T-JT^!;dO zXU}5WXLsv;{|->~*!Dr&TVcjEL)hJS3(Rq!bfb$Wr+&Cn@HkoMXNTGtK{Mnu9Tp`0 z=e`@2uKkq<1u*DEF#z)kCi~bRn`C?H<-3Wz`ssWt&i&muH4xNzwojseE$k-+mMn;X z7`?oq+@E&FdpMbXB3S3Pg6~25XS5`L*g;XaIe9Y%(XyxV4>G&QQCUrOboxcVn1iPN zl#99jiC-x~#zes>`mQEcY{W1A!0UK{Zd)l~oZ&hS(fUf|&5Ahg+@}hs_QAVgnfyPC z0;)ych2#RDv#<1OUtXtZHIbZ+QQ(U($c83!Xrq6z;x?=UYYbkJDSb3nY?}K>EB)Kf zs2kV^L?Sf)s?+GM|e!}XY_#Ofh+5V9xMHIy^sK*yHP;Yb3BIz=kab~Gi|V(btej;d0J|jh zUhZixs>1?Avdw%1zlP3=vAn!IkcC>nafv~|1!T1YCK2mVQmn=2fd&C1CC8qq@=dEJvF~-;*uJUv#+*C1y7IP7{S&N6p%(Y*S z!dYxqVYMmuo?DuFG2cuTcT;xEB$^9}RE_UGsci((aM^OwzFd&>fjTa7h&$~0xN@oH zzJ-{Vr5`%)R9eW6HYtyaMaCs@53$;SEvySq9e%Js4N?6b=}48So+M z{-o-#W5ennz4ph8>S-#k70ICM9+1uB{Y~Hz9dviHaHc|s^%lw*^ycaDizp6l|qY|GL5tR-~qD!VxU;ODDg*FlD3HnhaNgGgu6L@$F{U|XD1C#s|5hD zSgNr^eK;=_FZTKS2id_oh0lI&0$_$OGPo@x@$v!kxzrIPu974Awf(B?WbzC;3{8_! z&hx`^k2g1t_1NMGxOKlh?ZJhr2ykd^-%B)Aj(|*miADolX7pSNuuZ z@9OvqE+%l|Ub0{y!45g*p5TsjVVR3jQ&GWQ1w>-^zkgIZnqLSwMQf;;n3%+I7T&j{ z0$L+FP8Yc8epK%>Q%@K3`w5|CH>``U zGVS2d`&Lw3JgF9VMwlVuRh3T1$iPr`g~jD~2ed;8K>NhgQp&6UgGq-RYk*U8bJe_7 zJ<=e^#KgptB`TS_0E{mgPUHOMc$%@3vx}79`tamj?pEPlK zwA?p-28=Kje=eZX8#Xv)lf89V{PEJP_DA``tM$)DS6BuK);8L|F?NP( zmLP55*x+EEDS)!=vhS^WPnOOSoc+TBAOGe~*N%ggP;n?v?MboIZ@V_)lhuLTV^{of zTYUF&*_qVyW$+g+v{LGXgaiwQ&`GuR_JBPgizu!RII_13cy5EajNARSONS+pfU(P| z10(UT%1Lw42^rOa= zqN)mKfn(yClygv;sYi;TiC!*PHfvEWyMJV*{08zmNWM?BM>GA(5ptB6&C$$w) z5lzGApnphs2Y|*6D;Zfj2k&OI)M9ew50TX2e{O?4*6RX@_q||JOhVwf`NCO7XP8oAvNT~M;~5|3=I`TMxtz@p)X^% z=3E^v(l2$jhZj4txfBMDq;q#;8g4!t6B4OJVkdEAFn`i7#MfP26oMY2aj1l4tjBY( zeX->1;!!rQ!e=YT#h;89GV2QS71mjM^y9^PQ8$KP(f_*>A;E2bOrLJk(7s7{BV#QA zkhQ{&NG~|=ev7mt5A=8%wY5e|gmEfF!E1x}aA++~s0395-xQs8&#leZ*-nQi;T_8R z4iJfhlyA&}RaI4W4N1QarQmzyEUC0P2#miWD*6z=;*;1s3k*uKybgPCPi; zxy2hs2mSqf0@!?51O)UL=B(41&0*&ElB$lqHMB1kfc;cf!Z9xt3n{u;o-NB{$RH9O zu=7k~Y5JJ$;D~jh>o$8HDFSHNzJ?sGXBSxCd6|S?2!1fb)R2^v392?}li{#U0SDZw z1KNO5owVdDe4!)O9wdO(9GKOGy>l|e@2a9(_=+e+V8i>&+e_7QNm~5PZi@qsSgTv< z2#$G_(E!qi^^@KA2AZa`*?9>DQYk{Y_k}=eu5(C@&@~l1ktO72_|EPf88!gukQeFe zEkdaHKO(VfLd3~Ezs}|mF01h<;{m0hl_(pwVE_EyuPZ(o${tmsnq84YTKLKVGROrF zz*Gu6G_sGxA>II3@16O@W`;{42ZAakGy$dy#I^6mlfo;Z)!V5dfO>4kn0o@Na`}))!fk}&b$^SVC z`PKV|7WchR{=>@e0LT~tuyG9_ytTPIFK6V+D=P}-s9Gj-A8&~~_SZ5oFyekQ!iOB8 zQcpEf>$bmQWrZ@b53Y3NdaehFXSeQ9ubydKUAjLV-|gjmzS>Q_1bLL+ovf=|t+;76 z)SZAAy}+g|pP4+A1|l$_M(^pu0Dw#rE`^^1WaY!N&Hql+!p_bxZ8snqyk@&-z~MX+ z7gS|(1y~CJSv_X?oYzQYuT#_{~ zSr|q?9W-R&jKrdP2*+C$=G&DsCnpvZf^G)>0xwq@)+@GgNC`**xS1Rn;Ly2_9biTM z{LRB>cd|o6OUo^bBnu7X{~Ntr_54#feRt3X9B-)mY9Pqd)a-TqaJMx0JF0PfRds05BmS7v8P7_m#;JjArePHX*Mm9s{k66h-_^rIho(1}`u?JbKv3t#m!oE~jRM<7Wk)Yupy7*{EY(WH2_3=)?aq;Yl;7%8GDF9*!P~^0$ zfulNLZJ_aKYN`X%hG-HHdJKoUSTdUjAiOML*iYrg?AMU$_El6?o@BS|XTQiEpC4o5 z@U?Pv{qgS@Lk~w~wg`4Ft-&HLD=RzwTc)a}mX^u21`tqthHE+ESekgi{8hz|9q*oU z(0X^=5?Nv#|7>-jSb$T{Y#@IWfB!inW8W)bWJI|My$6U|e8)k=5>x$10Zg*5$sO`S zL5=y8@y{bfr%){Hu)w85gXfc|@>Brr3_-L&#e%jj3=GGwOvJMSK%XvQ-p-~cA2w4y ztS1$MAfm!>2az^|{3y=0^Vprp!NA0XmS7bEBf;*Zq1%OH2sy!49Dl1K+ZRC>(yAK_ z*y?D0wsL>EtZQyIbii9ERwC_ib3d7kC}H#}6tmu$y2#Y~oyLvz99y&4vUQKU3WlG- zDX;e14J`)7h?nrtip_|^j*`>Z4Ct^17utxB$6|mp1*xCAt-P?}vk_5&E^>1AnfIP| zE~n`ENVzd#(F0snq``?#qrdoFMn^;Db;q9WaOo`ZAbzp;_S$qg5f9=glzFkI z4e{8(9@oxqe!X#r`evI_ZQJVmuP7V5<@gR34F}k`x{BrQssb(LU<;Nh%~45c7l!T- zUJ-^RU;IK>qi3O{au-Q)`x{b4Tma|_n8RG?1v0C&498w)bX2!MV4ZV>H~`AnJBJ2X zG)JV60a9d1Z?6pPKXR&U97&9ZYM?DYRXap8&_^9G3QYW{3w$;N*fmlJ=kjdZ0d0|c zGwNgLV*}?XdWk>#mgVy|>8Tm_uxE?F7-8%Sf$FL58ES2fTqZqasN6IkZ9FC&kz79? z&MY`Rm0rBSY8lc$dU0u9; znm*^W1UpKX)3=5F4SgDd?&eKo9ahtbpp_I_t8aXPeO>d5$7N`HN^KRx4;@f`AiD1d&@Xxj{DGGc-ajT@o-)a8E_2F0mF{ zoV#7fwHXr=lO?k&AN>lcI0igGRc5oOtZL~vIy0uJ-&7Dy2FMI(P)4vkvvXACUFQd* z3Qw!~?(W`h6hR3)t~akR`-*@_1hhz9o&|&!)AdH9``++?sK-Xi!&KMs}fz zoneXDQ#V}hpFnR_Q)W%_bfQqvaV;xvdo!k%DDhpKeoB&dfAkUZyfbq{4?iTyYjdfb0OKRm(TB)34G{>*W$OW6ZsnwIs@HXmc)uQ%>D)iu z7b8*i9tb#E5m2(hJgWJi4`~dh+3}+^ z`8Q6e2PxEUSTOywj9=}}uP95Lk}QRO< z@iY&#$A!MdyLy3pmZQus=I8{PNkEuJA zJ1Q+Bz~)DB@t%DRG6=d#5WCQc=s#t4cL|x?j{s#0a{agj=jrJ`LvV60RkNXa zqmi6VAVpNo;Kj(*7wEXqE4TUh>iz;}ON=sgy2#-+*Pg zuz21@8%Vwx6-w^#ovukLJH|k*P;4~yKVwzHP+R|F4tW5ezSS^HZR>}$O{K()7&VTQ z(NCwQv~<6c8;va!SXTw2E(n4VQuoz~p~Z$%XU2qp<0)Fq?T0f#@L%f7#3=cj{4A(z zePKBsCS^~buGFCKuSRWu$WpLQ)E~;WV+cKx#d;HRcd51XdjurWt26dEHXgWR>o>TVYT`zRe4 z;kwWnsVV2BSc0aqIE7=c!n4I!6jdx^78HGFHznIO4v?FcSOxX`AyNg{*A6%d-aR!T zdC3syb>n*&?8~1oUyfV}2h^w4+2mY??hJf4*j+!rkWq4q*{}+?83Z;4kIONScLZZZ zp!<9XqK(UiK}{@VZ(+bfC%=)=$Z6kht^`*vV#i^Ux@pN}=0?tOVIA{iuU0I(Z|%|v z^@Wg=7`Ms0f$&_C&T&sgI8yp77I2H|I38#Mo4l}>beugOe>3#zaWQ9}3OY%N(d*Oo z1~+Kk)j1vxNipwLDs<;?kGW)$1Jos==Fe51-qNzj!Da1*3OEkboVL&*@8TG&OEFe= zdUb>9jpH3SQ%Zj6RFzyRe)Y~{)*7cOEg4c-39s!%_3_v_IjVYeQ71V$p-;7LL%GD1 z#w$0nZ8U`5`7JyD?bFvVbTg3|uoqvh^=gwNCtZblmv;$mh0*SwNuNvsVBc1DNn%~g zoPw$)i%!!_kUcfV93g~dXR(0OX8fyAN>;MM-%C&Y>3zBvCEa}RJJGl@DD7``b?`+jn=P=4>$!x}X)67iXe?}FU=N~@|8NjQuZo|*O}dY2*~{n-H5 z0kjSGv87*!`<$$`HG|U4ihuK?*kyU9Kf;cYmmvQ5|tO+ znibVvW1hX6uE#4v6o4cUca?ay|E)T9ax)knQKk#6#sf%tmhxKx^y&^MRX68}QO_XY zLhXDD0QxLzmG>yzw_mv}QHo1SVufs6Txyup(@Fp_@c!nnto5(zumaQeLvVwy=yw{( z($!MA7Spprcuj-kOTov-Hwj$dv7IcCwYaYQM@oq_3jjq5QoAvg#^up!{fY475b9Pj=cg&mkC8V+&F~*0>HThn= z`fk*q>c3GJz-K*1*%RQX_gtp!lMDwKxV97MjMUSfl0uaKGa0Tcbk}Cw>+JWNrmx%2 z_iWZdEzaxh7D!|)IyySM?z`$8ZdC@ga!!B9o=b3fl3@fxhRu+bnr`MzgR)h#gN7}$$m$f)=UaObai1v<#-XOBu4CLE)fQfnkhIU5v^_&|3 z0zw|qP5sBRgbLTVhzO@&)`sYq+8YBEm#*|)I;=EKp5VT~)I(_Gw@EXuG2A5r8vQS- zzjU*e!lL!tZn{%cyJ_rSo_`^x8iL6F>rUpY4j;Ee%|J>~Z}m1+Kq)G3U3t8dcKHvN z=R(zSBBKBNc(vfBJM!@BRm0r(< zwjjKW;4drc(4j92hurV49?Q$hY$i!V6r(2%y|-%3x)3fHt;e!h^fJdHv0uy_bS?ZL zdKPNe1hP0SM{~sdfJctBWH+)my!<|F2}B72_XCIwu%kN^5iJA69(4Vq>LFl%*%qkAqM9xW#eVKM`jM{F|Nh#%*qfc|Ms zC8DW#FxPCQm_*GT$I^3-C+A(9`$26}T~}8%=wavYnfY5TuPA$js;I}%ZEG5wTtQ{8 zx2-r@%4Qs3P@XWcrsiip> z2ai4Q{jShJfH1J@9k{|YsoTyBNpDwsl7@9?$-(}OzNe$UaJHw#3*%YKWlPoYMPWf=l zbi7sTaiDwg=Oc)LGROOmZ1W{Ftu4PM`SA>Q8&FI<*fxZXHmEB>>F>Hl3 zQbC)$?)iXzyg$cH7xIEXTJ(|tQxefX6q*%Z%PfmuU}K-}Fy)k>f_y@MsJ`gv_@|!2 z1a9>Cj-7n@Cpi_B(JuSbN|^yNWTl`Xk7uny@6H?*7NQjF+VR0WDO{l!wNMJ$>>Y~? z5PnO!r>tLK`CMhJlX|&Z`7T8mcVQ4ju+Do?(f1hY>8hjNy_eeaGMh9~7z*gT?GwMxXByH~gXS=|^ z1K~1B%JJ=6eV+J#njiL39Qp@O0mFlD=s!YQ&)0AEVinQ`n@2R@Q#_qxQl*k)@Apxx z;AjdFx2oLTa}Q!lf*=%)0v9y1Gjp0#v9sr}>tC*) zgTt4_T}VMIdHl5a7On9=m~A8UDcJud!BHHi?C;ynj-v*m*tx;Rz+m3(d?huCYo8>cCp!W~z_#brEi7W4 z04*r{gLn*b^5y?xD4-|Wyd0mSmuH@2V#^;gd}vhL-a<69(RCtd%18)R-TUtn-%scD zDiho{JJK53+A^5_*=Y#!(%G*Iy-w1F$U}h(uovx2s#(l>nK+LmfEI`=LiT}CB*#Pa z-7n$He=la9UGxuxIoCD;0UxV{W)v7G@Ww+cqlYda7#l=*^93scQ#vNi?r)qdD57g8 zeS5q4t-YRCb%m#JCa`|_6b8Z>*8s-5MZGG-{5ZF&^nS8&%Lw^jo7-t0 zs@c1ORWdvw`R+W+34Ilyo?nf1~FDcqL*(QldK5$x~I_on+GnnI|;12LXCW@bgvNE=Im zq5AYXmB}ou;NWPN&kgxd6D~mGQB*n-Un2Ok?q@s`hW;3Q~U-LL>kXrnGo}!n=@#4kI`L`=1`eNVfcEY z(X(SM;(@U38jhKD$^psuGcz6;)`|LWcGLRoXm+2hRy93lv&w;}t!%F~Ti9a)$-B=o zzJyFpOhg#}avJ=JPC}65g=7pDlD8pDhFrN;o^3AS-)|-hkhW7UsceQ4E59+1j;i5K z%~||sn$ymc081}il!W_OX>Q;vO-Va>5j@5+PJ-@29w}&?hX@yfMPqyl%D?qRSbn#H zD{v0mVj+%knPh+V8*DtgOOX(=fAex$MVed4-9Mj#\n", "\n", - "In this example, we will demonstrate the SAG workflow with FedAvg using CIFAR10 dataset. \n", + "In this example, we will demonstrate the SAG workflow with FedAvg using the CIFAR10 dataset.\n", "\n", - "Both Job Lifecycle and training workflow are controlled on the **server side**, we will just use the existing available SAG controller availalbe in NVFLARE. \n", + "Both Job Lifecycle and training workflow are controlled on the server side; we will just use the existing available SAG controller available in NVFLARE.\n", + "\n", + "For client-side training code, we will leverage the new DL to FL Client API.\n", + "\n", + "First, let's look at the FedAvg Algorithm and SAG Workflow.\n", + "\n", + "\n", + "## Scatter and Gather (SAG)\n", + "\n", + "Scatter and Gather are part of the Message Passing Interface (MPI). [MPI](https://en.wikipedia.org/wiki/Message_Passing_Interface) is a standardized and portable message-passing standard designed to function on parallel computing architectures. MPI consists of some [collective communication routines](https://mpitutorial.com/tutorials/mpi-broadcast-and-collective-communication/), such as broadcast, scatter, and gather. [Scatter and Gather](https://mpitutorial.com/tutorials/mpi-scatter-gather-and-allgather/) are widely used for federated learning in aggregation algorithms such as Fed Average.\n", + "\n", + "\"scatter\"\"gather\"\n", "\n", - "For client side training code, we will leverage new DL to FL **Client API**\n", "\n", - "First, Let's look at the FedAvg Algorithm and SAG Workflow. \n", "\n", "## FedAvg with SAG\n", + "[FedAvg](https://nvflare.readthedocs.io/en/main/programming_guide/controllers/scatter_and_gather_workflow.html) is a workflow that leverage MPI Scatter and Gather. You can see one round of training in such workflow.\n", + "\n", + "\"FedAvg\"\n", + "\n", "\n", "\"FedAvg\" \"Scatter\n", "\n", @@ -91,12 +104,11 @@ "source": [ "## Job Folder and Configurations\n", "\n", + " \n", + "Now we need to set up the configurations for the server and clients and construct the Job folder NVFLARE needs to run. We can do this using NVFLARE job CLI. You can study the [Job CLI tutorials](https://github.com/NVIDIA/NVFlare/blob/main/examples/tutorials/job_cli.ipynb) later with all the details. But for now, you can just use the following commands to find out the available job templates.\n", "\n", - "Now we need to setup the configurations for server and clients and constructure Job folder NVFLARE needed to run. We can do this using NVFLARE job CLI. You can study the [Job CLI tutorials](https://github.com/NVIDIA/NVFlare/blob/main/examples/tutorials/job_cli.ipynb) later with all the details. But for now, you can just use the following commands\n", - "\n", - "* Find out the available job templates\n", - "\n", - "We need to set the job templates directory, so the job cli commands can find the job templates. If have already set NVFLARE_HOME=``` ```then, you can skipt the folllowing step. \n" + "We need to set the job templates directory so the job CLI commands can find the job templates. If you have already set `NVFLARE_HOME` to ``, then you can skip the following step.\n", + "\n" ] }, { @@ -130,7 +142,7 @@ "source": [ "* Create job folder and initial configs\n", "\n", - "The template **'sag_pt'** seems to fit our needs: SAG with pytorch, using client API. Lets create a job folder with this template initially without specifying the code location, just see what's needs to be changed" + "The template **'sag_pt'** seems to fit our needs: SAG with PyTorch, using the client API. Let's create a job folder with this template initially without specifying the code location, just to see what needs to be changed.\n" ] }, { @@ -182,10 +194,9 @@ "id": "dbb79b5c-f97f-472f-91a5-5a5175fb9759", "metadata": {}, "source": [ - "* Create job folder with all the configs\n", + "* Create a job folder with all the configurations.\n", "\n", - "Let's change the num_rounds = 5, script = train.py, min_clients = 2 for meta.conf. We also like to change the arguments for train.py \n", - "dataset_path=CIFAR10_ROOT, batch_size=6, num_workers = 2. Here dataset_path is actually not changed, but we just want to show you could change. " + "Let's change the `num_rounds` to 5, `script` to `train.py`, and `min_clients` to 2 in `meta.conf`. We also want to change the arguments for `train.py`: `dataset_path=CIFAR10_ROOT`, `batch_size=6`, `num_workers=2`. Note that the `dataset_path` is not actually changed, but we just want to show you that it could be changed.\n" ] }, { @@ -257,9 +268,9 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "nvflare_example", "language": "python", - "name": "python3" + "name": "nvflare_example" }, "language_info": { "codemirror_mode": { diff --git a/examples/hello-world/step-by-step/cifar10/sag_model_learner/sag_model_learner.ipynb b/examples/hello-world/step-by-step/cifar10/sag_model_learner/sag_model_learner.ipynb index 081ad00674..261275427c 100644 --- a/examples/hello-world/step-by-step/cifar10/sag_model_learner/sag_model_learner.ipynb +++ b/examples/hello-world/step-by-step/cifar10/sag_model_learner/sag_model_learner.ipynb @@ -19,7 +19,7 @@ "\n", "Key Concepts:\n", "- Learning\n", - " - `FLModel` object defines structure to containe essential information about the learning task, such as `params`, `metrics`, `meta`, etc.\n", + " - `FLModel` object defines structure to contain essential information about the learning task, such as `params`, `metrics`, `meta`, etc.\n", " - learning logic implemented in `train()` and `validate` methods, which both receive and send an `FLModel` object\n", " - return requested model via `get_model()`\n", "- Lifecycle\n", @@ -216,9 +216,9 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3", + "display_name": "nvflare_example", "language": "python", - "name": "python3" + "name": "nvflare_example" }, "language_info": { "codemirror_mode": { @@ -230,7 +230,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.10" + "version": "3.8.16" } }, "nbformat": 4, diff --git a/examples/tutorials/setup_poc.ipynb b/examples/tutorials/setup_poc.ipynb index 26b99210c1..d1a1aa0007 100644 --- a/examples/tutorials/setup_poc.ipynb +++ b/examples/tutorials/setup_poc.ipynb @@ -7,7 +7,7 @@ "source": [ "# Set Up NVFLARE in POC Mode\n", "\n", - "[POC mode](https://nvflare.readthedocs.io/en/main/user_guide/poc_command.html) allows users to test the features of a full FLARE deployment on a single machine, without the overhead of a true distributed deployment.\n", + "[POC mode](https://nvflare.readthedocs.io/en/main/user_guide/nvflare_cli/poc_command.html) allows users to test the features of a full FLARE deployment on a single machine, without the overhead of a true distributed deployment.\n", "\n", "Compared to the FL Simulator, where the job run is automated on a single system, POC mode allows you to establish and connect distinct server and client \"systems\" which can then be orchestrated using the FLARE Console. This can be useful in preparation for a distributed deployment.\n", "\n", @@ -628,23 +628,24 @@ "source": [ "### Support Homomorphic Encryption (HE)\n", "\n", - "To support HE, we need the provision process to generates Tenseal homomorphic encryption context for server and client and writes them to server and client\n", - "participant folders see [provision context](https://nvflare.readthedocs.io/en/main/programming_guide/provisioning_system.html#provision-context). This is achieved by Provision builder, in particular for HE, HEBuilder. Instead of manaully add HEBuilder to project.yml file, one can use ```-he``` in poc command\n", + "To support HE, we need the provision process to generate Tenseal homomorphic encryption context for the server and client and write them to the server and client participant folders. See [provision context](https://nvflare.readthedocs.io/en/main/programming_guide/provisioning_system.html#provision-context). This is achieved by the Provision builder, specifically for HE, HEBuilder. Instead of manually adding HEBuilder to the `project.yml` file, one can use `-he` in the POC command.\n", + "\n", + "For example, if we use the above command with HE, we can write as\n", "\n", - "For example, if we use above command with HE, we can write as\n", "\n", "```\n", "echo 'y' | nvflare poc prepare -c hospital_1 hospital_2 -d 'nvflare/nvflare' -he\n", "\n", "```\n", - "But before you run the command, you must have the correct dependency. NVFLARE uses Tenseal library as HE dependency. By default it is optional dependency. \n", - "To use HE, you could install it with \n", + "\n", "\n", "```\n", " pip install nvflare[HE]\n", + " \n", "```\n", - "> Note\n", - " * Tenseal is not avaiable in Mac OS\n" + "\n", + "\n", + "> note: Tenseal is not avaiable in Mac OS\n" ] }, { From e4f8d417d1d8593fe0f658559269fd54265b7067 Mon Sep 17 00:00:00 2001 From: Yuhong Wen Date: Tue, 16 Jan 2024 18:26:09 -0500 Subject: [PATCH 05/39] Fixed the client_executor improper lock use. (#2282) Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> --- nvflare/private/fed/client/client_executor.py | 212 +++++++++--------- 1 file changed, 106 insertions(+), 106 deletions(-) diff --git a/nvflare/private/fed/client/client_executor.py b/nvflare/private/fed/client/client_executor.py index eae4b8abb0..c775f7b8b9 100644 --- a/nvflare/private/fed/client/client_executor.py +++ b/nvflare/private/fed/client/client_executor.py @@ -21,11 +21,12 @@ import time from abc import ABC, abstractmethod -from nvflare.apis.fl_constant import AdminCommandNames, RunProcessKey +from nvflare.apis.fl_constant import AdminCommandNames, RunProcessKey, SystemConfigs from nvflare.apis.resource_manager_spec import ResourceManagerSpec from nvflare.fuel.common.exit_codes import PROCESS_EXIT_REASON, ProcessExitCode from nvflare.fuel.f3.cellnet.core_cell import FQCN from nvflare.fuel.f3.cellnet.defs import MessageHeaderKey, ReturnCode +from nvflare.fuel.utils.config_service import ConfigService from nvflare.private.defs import CellChannel, CellChannelTopic, JobFailureMsgKey, new_cell_message from nvflare.private.fed.utils.fed_utils import get_return_code from nvflare.security.logging import secure_format_exception, secure_log_traceback @@ -138,6 +139,10 @@ def __init__(self, client, startup): self.run_processes = {} self.lock = threading.Lock() + self.job_query_timeout = ConfigService.get_float_var( + name="job_query_timeout", conf=SystemConfigs.APPLICATION_CONF, default=5.0 + ) + def start_app( self, client, @@ -216,10 +221,9 @@ def start_app( thread.start() def notify_job_status(self, job_id, job_status): - with self.lock: - run_process = self.run_processes.get(job_id) - if run_process: - run_process[RunProcessKey.STATUS] = job_status + run_process = self.run_processes.get(job_id) + if run_process: + run_process[RunProcessKey.STATUS] = job_status def check_status(self, job_id): """Checks the status of the running client. @@ -231,9 +235,8 @@ def check_status(self, job_id): A client status message """ try: - with self.lock: - process_status = self.run_processes.get(job_id, {}).get(RunProcessKey.STATUS, ClientStatus.NOT_STARTED) - return get_status_message(process_status) + process_status = self.run_processes.get(job_id, {}).get(RunProcessKey.STATUS, ClientStatus.NOT_STARTED) + return get_status_message(process_status) except Exception as e: self.logger.error(f"check_status execution exception: {secure_format_exception(e)}.") secure_log_traceback() @@ -249,23 +252,23 @@ def get_run_info(self, job_id): A dict of run information. """ try: - with self.lock: - data = {} - fqcn = FQCN.join([self.client.client_name, job_id]) - request = new_cell_message({}, data) - return_data = self.client.cell.send_request( - target=fqcn, - channel=CellChannel.CLIENT_COMMAND, - topic=AdminCommandNames.SHOW_STATS, - request=request, - optional=True, - ) - return_code = return_data.get_header(MessageHeaderKey.RETURN_CODE) - if return_code == ReturnCode.OK: - run_info = return_data.payload - return run_info - else: - return {} + data = {} + fqcn = FQCN.join([self.client.client_name, job_id]) + request = new_cell_message({}, data) + return_data = self.client.cell.send_request( + target=fqcn, + channel=CellChannel.CLIENT_COMMAND, + topic=AdminCommandNames.SHOW_STATS, + request=request, + optional=True, + timeout=self.job_query_timeout, + ) + return_code = return_data.get_header(MessageHeaderKey.RETURN_CODE) + if return_code == ReturnCode.OK: + run_info = return_data.payload + return run_info + else: + return {} except Exception as e: self.logger.error(f"get_run_info execution exception: {secure_format_exception(e)}.") secure_log_traceback() @@ -281,23 +284,23 @@ def get_errors(self, job_id): A dict of error information. """ try: - with self.lock: - data = {"command": AdminCommandNames.SHOW_ERRORS, "data": {}} - fqcn = FQCN.join([self.client.client_name, job_id]) - request = new_cell_message({}, data) - return_data = self.client.cell.send_request( - target=fqcn, - channel=CellChannel.CLIENT_COMMAND, - topic=AdminCommandNames.SHOW_ERRORS, - request=request, - optional=True, - ) - return_code = return_data.get_header(MessageHeaderKey.RETURN_CODE) - if return_code == ReturnCode.OK: - errors_info = return_data.payload - return errors_info - else: - return None + data = {"command": AdminCommandNames.SHOW_ERRORS, "data": {}} + fqcn = FQCN.join([self.client.client_name, job_id]) + request = new_cell_message({}, data) + return_data = self.client.cell.send_request( + target=fqcn, + channel=CellChannel.CLIENT_COMMAND, + topic=AdminCommandNames.SHOW_ERRORS, + request=request, + optional=True, + timeout=self.job_query_timeout, + ) + return_code = return_data.get_header(MessageHeaderKey.RETURN_CODE) + if return_code == ReturnCode.OK: + errors_info = return_data.payload + return errors_info + else: + return None except Exception as e: self.logger.error(f"get_errors execution exception: {secure_format_exception(e)}.") secure_log_traceback() @@ -310,17 +313,16 @@ def reset_errors(self, job_id): job_id: the job_id """ try: - with self.lock: - data = {"command": AdminCommandNames.RESET_ERRORS, "data": {}} - fqcn = FQCN.join([self.client.client_name, job_id]) - request = new_cell_message({}, data) - self.client.cell.fire_and_forget( - targets=fqcn, - channel=CellChannel.CLIENT_COMMAND, - topic=AdminCommandNames.RESET_ERRORS, - message=request, - optional=True, - ) + data = {"command": AdminCommandNames.RESET_ERRORS, "data": {}} + fqcn = FQCN.join([self.client.client_name, job_id]) + request = new_cell_message({}, data) + self.client.cell.fire_and_forget( + targets=fqcn, + channel=CellChannel.CLIENT_COMMAND, + topic=AdminCommandNames.RESET_ERRORS, + message=request, + optional=True, + ) except Exception as e: self.logger.error(f"reset_errors execution exception: {secure_format_exception(e)}.") @@ -332,41 +334,41 @@ def abort_app(self, job_id): Args: job_id: the job_id """ - with self.lock: - # When the HeartBeat cleanup process try to abort the worker process, the job maybe already terminated, - # Use retry to avoid print out the error stack trace. - retry = 1 - while retry >= 0: - process_status = self.run_processes.get(job_id, {}).get(RunProcessKey.STATUS, ClientStatus.NOT_STARTED) - if process_status == ClientStatus.STARTED: - try: + # When the HeartBeat cleanup process try to abort the worker process, the job maybe already terminated, + # Use retry to avoid print out the error stack trace. + retry = 1 + while retry >= 0: + process_status = self.run_processes.get(job_id, {}).get(RunProcessKey.STATUS, ClientStatus.NOT_STARTED) + if process_status == ClientStatus.STARTED: + try: + with self.lock: child_process = self.run_processes[job_id][RunProcessKey.CHILD_PROCESS] - data = {} - fqcn = FQCN.join([self.client.client_name, job_id]) - request = new_cell_message({}, data) - self.client.cell.fire_and_forget( - targets=fqcn, - channel=CellChannel.CLIENT_COMMAND, - topic=AdminCommandNames.ABORT, - message=request, - optional=True, - ) - self.logger.debug("abort sent to worker") - t = threading.Thread(target=self._terminate_process, args=[child_process, job_id]) - t.start() - t.join() - break - except Exception as e: - if retry == 0: - self.logger.error( - f"abort_worker_process execution exception: {secure_format_exception(e)} for run: {job_id}." - ) - secure_log_traceback() - retry -= 1 - time.sleep(5.0) - else: - self.logger.info(f"Client worker process for run: {job_id} was already terminated.") + data = {} + fqcn = FQCN.join([self.client.client_name, job_id]) + request = new_cell_message({}, data) + self.client.cell.fire_and_forget( + targets=fqcn, + channel=CellChannel.CLIENT_COMMAND, + topic=AdminCommandNames.ABORT, + message=request, + optional=True, + ) + self.logger.debug("abort sent to worker") + t = threading.Thread(target=self._terminate_process, args=[child_process, job_id]) + t.start() + t.join() break + except Exception as e: + if retry == 0: + self.logger.error( + f"abort_worker_process execution exception: {secure_format_exception(e)} for run: {job_id}." + ) + secure_log_traceback() + retry -= 1 + time.sleep(5.0) + else: + self.logger.info(f"Client worker process for run: {job_id} was already terminated.") + break self.logger.info("Client worker process is terminated.") @@ -405,25 +407,23 @@ def abort_task(self, job_id): Args: job_id: the job_id """ - with self.lock: - process_status = self.run_processes.get(job_id, {}).get(RunProcessKey.STATUS, ClientStatus.NOT_STARTED) - if process_status == ClientStatus.STARTED: - data = {"command": AdminCommandNames.ABORT_TASK, "data": {}} - fqcn = FQCN.join([self.client.client_name, job_id]) - request = new_cell_message({}, data) - self.client.cell.fire_and_forget( - targets=fqcn, - channel=CellChannel.CLIENT_COMMAND, - topic=AdminCommandNames.ABORT_TASK, - message=request, - optional=True, - ) - self.logger.debug("abort_task sent") + process_status = self.run_processes.get(job_id, {}).get(RunProcessKey.STATUS, ClientStatus.NOT_STARTED) + if process_status == ClientStatus.STARTED: + data = {"command": AdminCommandNames.ABORT_TASK, "data": {}} + fqcn = FQCN.join([self.client.client_name, job_id]) + request = new_cell_message({}, data) + self.client.cell.fire_and_forget( + targets=fqcn, + channel=CellChannel.CLIENT_COMMAND, + topic=AdminCommandNames.ABORT_TASK, + message=request, + optional=True, + ) + self.logger.debug("abort_task sent") def _wait_child_process_finish(self, client, job_id, allocated_resource, token, resource_manager, workspace): self.logger.info(f"run ({job_id}): waiting for child worker process to finish.") - with self.lock: - child_process = self.run_processes.get(job_id, {}).get(RunProcessKey.CHILD_PROCESS) + child_process = self.run_processes.get(job_id, {}).get(RunProcessKey.CHILD_PROCESS) if child_process: child_process.wait() @@ -452,13 +452,13 @@ def _wait_child_process_finish(self, client, job_id, allocated_resource, token, resource_manager.free_resources( resources=allocated_resource, token=token, fl_ctx=client.engine.new_context() ) - self.run_processes.pop(job_id, None) + with self.lock: + self.run_processes.pop(job_id, None) self.logger.debug(f"run ({job_id}): child worker resources freed.") def get_status(self, job_id): - with self.lock: - process_status = self.run_processes.get(job_id, {}).get(RunProcessKey.STATUS, ClientStatus.STOPPED) - return process_status + process_status = self.run_processes.get(job_id, {}).get(RunProcessKey.STATUS, ClientStatus.STOPPED) + return process_status def get_run_processes_keys(self): with self.lock: From 9a7942ef01a9f0ede8d02021113e7f17378cb990 Mon Sep 17 00:00:00 2001 From: Ziyue Xu <71786575+ZiyueXu77@users.noreply.github.com> Date: Wed, 17 Jan 2024 00:13:59 -0500 Subject: [PATCH 06/39] add notebook for gnn examples (#2289) Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> --- examples/advanced/gnn/README.md | 10 +- .../advanced/gnn/code/graphsage_finance_fl.py | 2 +- .../gnn/code/graphsage_finance_local.py | 2 +- examples/advanced/gnn/gnn_examples.ipynb | 336 ++++++++++++++++++ examples/advanced/gnn/requirements.txt | 1 - 5 files changed, 343 insertions(+), 8 deletions(-) create mode 100644 examples/advanced/gnn/gnn_examples.ipynb diff --git a/examples/advanced/gnn/README.md b/examples/advanced/gnn/README.md index 7983e528d7..3a743431ce 100644 --- a/examples/advanced/gnn/README.md +++ b/examples/advanced/gnn/README.md @@ -55,7 +55,7 @@ python3 code/graphsage_protein_local.py --client_id 2 ``` Then, we create NVFlare job based on GNN template. ``` -nvflare job create -force -j "./jobs/gnn_protein" -w "sag_gnn" -sd "code" \ +nvflare job create -force -j "/tmp/nvflare/jobs/gnn_protein" -w "sag_gnn" -sd "code" \ -f app_1/config_fed_client.conf app_script="graphsage_protein_fl.py" app_config="--client_id 1 --epochs 10" \ -f app_2/config_fed_client.conf app_script="graphsage_protein_fl.py" app_config="--client_id 2 --epochs 10" \ -f app_server/config_fed_server.conf num_rounds=7 key_metric="validation_f1" model_class_path="torch_geometric.nn.GraphSAGE" components[0].args.model.args.in_channels=50 components[0].args.model.args.hidden_channels=64 components[0].args.model.args.num_layers=2 components[0].args.model.args.out_channels=64 @@ -66,11 +66,11 @@ For server configs, we set the number of rounds for federated training, the key With the produced job, we run the federated training on both clients via FedAvg using NVFlare Simulator. ``` -nvflare simulator -w /tmp/nvflare/gnn/protein_fl_workspace -n 2 -t 2 ./jobs/gnn_protein +nvflare simulator -w /tmp/nvflare/gnn/protein_fl_workspace -n 2 -t 2 /tmp/nvflare/jobs/gnn_protein ``` #### Financial Transaction Classification -We first download the Elliptic++ dataset to `data` folder. In this example, we will use the following three files: +We first download the Elliptic++ dataset to `/tmp/nvflare/datasets/elliptic_pp` folder. In this example, we will use the following three files: - `txs_classes.csv`: transaction id and its class (licit or illicit) - `txs_edgelist.csv`: connections for transaction ids - `txs_features.csv`: transaction id and its features @@ -82,14 +82,14 @@ python3 code/graphsage_finance_local.py --client_id 2 ``` Similarly, we create NVFlare job based on GNN template. ``` -nvflare job create -force -j "./jobs/gnn_finance" -w "sag_gnn" -sd "code" \ +nvflare job create -force -j "/tmp/nvflare/jobs/gnn_finance" -w "sag_gnn" -sd "code" \ -f app_1/config_fed_client.conf app_script="graphsage_finance_fl.py" app_config="--client_id 1 --epochs 10" \ -f app_2/config_fed_client.conf app_script="graphsage_finance_fl.py" app_config="--client_id 2 --epochs 10" \ -f app_server/config_fed_server.conf num_rounds=7 key_metric="validation_auc" model_class_path="pyg_sage.SAGE" components[0].args.model.args.in_channels=165 components[0].args.model.args.hidden_channels=256 components[0].args.model.args.num_layers=3 components[0].args.model.args.num_classes=2 ``` And with the produced job, we run the federated training on both clients via FedAvg using NVFlare Simulator. ``` -nvflare simulator -w /tmp/nvflare/gnn/finance_fl_workspace -n 2 -t 2 ./jobs/gnn_finance +nvflare simulator -w /tmp/nvflare/gnn/finance_fl_workspace -n 2 -t 2 /tmp/nvflare/jobs/gnn_finance ``` ### Results diff --git a/examples/advanced/gnn/code/graphsage_finance_fl.py b/examples/advanced/gnn/code/graphsage_finance_fl.py index b2d8fa8a20..65ec991486 100644 --- a/examples/advanced/gnn/code/graphsage_finance_fl.py +++ b/examples/advanced/gnn/code/graphsage_finance_fl.py @@ -37,7 +37,7 @@ def main(): parser.add_argument( "--data_path", type=str, - default="./data", + default="/tmp/nvflare/datasets/elliptic_pp", ) parser.add_argument( "--epochs", diff --git a/examples/advanced/gnn/code/graphsage_finance_local.py b/examples/advanced/gnn/code/graphsage_finance_local.py index 351f9d4c1e..51404c3be2 100644 --- a/examples/advanced/gnn/code/graphsage_finance_local.py +++ b/examples/advanced/gnn/code/graphsage_finance_local.py @@ -34,7 +34,7 @@ def main(): parser.add_argument( "--data_path", type=str, - default="./data", + default="/tmp/nvflare/datasets/elliptic_pp", ) parser.add_argument( "--epochs", diff --git a/examples/advanced/gnn/gnn_examples.ipynb b/examples/advanced/gnn/gnn_examples.ipynb new file mode 100644 index 0000000000..54ab3ede98 --- /dev/null +++ b/examples/advanced/gnn/gnn_examples.ipynb @@ -0,0 +1,336 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cada310b-e776-4b9a-aabe-f111c31efcc2", + "metadata": { + "tags": [] + }, + "source": [ + "# Federated GNN on Graph Dataset using Inductive Learning" + ] + }, + { + "cell_type": "markdown", + "id": "0653cbf2-92f2-4a22-8317-69cfb0266e92", + "metadata": {}, + "source": [ + "## Introduction to GNN, Tasks, and federated GNN via Inductive Learning\n", + "### GNN\n", + "This example shows how to train a classification model using Graph Neural Network (GNN). GNNs show a promising future in research and industry, with potential applications in various domains, including social networks, e-commerce, recommendation systems, and more.\n", + "GNNs excel in learning, modeling, and leveraging complex relationships within graph-structured data. They combine local and global information, incorporate structural knowledge, adapt to diverse tasks, handle heterogeneous data, support transfer learning, scale for large graphs, offer interpretable insights, and achieve impressive performance. \n", + "\n", + "### Tasks\n", + "In this example, we provide two tasks:\n", + "1. **Protein Classification**:\n", + "The aim is to classify protein roles based on their cellular functions from gene ontology. The dataset we are using is PPI\n", + "([protein-protein interaction](http://snap.stanford.edu/graphsage/#code)) graphs, where each graph represents a specific human tissue. Protein-protein interaction (PPI) dataset is commonly used in graph-based machine-learning tasks, especially in the field of bioinformatics. This dataset represents interactions between proteins as graphs, where nodes represent proteins and edges represent interactions between them.\n", + "2. **Financial Transaction Classification**:\n", + "The aim is to classify whether a given transaction is licit or illicit. For this financial application, we use the [Elliptic++](https://github.com/git-disl/EllipticPlusPlus) dataset. It consists of 203k Bitcoin transactions and 822k wallet addresses to enable both the detection of fraudulent transactions and the detection of illicit addresses (actors) in the Bitcoin network by leveraging graph data. For more details, please refer to this [paper](https://arxiv.org/pdf/2306.06108.pdf).\n", + "\n", + "\n", + "### Federated GNN via Inductive Learning\n", + "Both tasks are for node classification. We used the inductive representation learning method [GraphSAGE](https://arxiv.org/pdf/1706.02216.pdf) based on [Pytorch Geometric](https://github.com/pyg-team/pytorch_geometric)'s examples. \n", + "[Pytorch Geometric](https://pytorch-geometric.readthedocs.io/en/latest/) is a library built upon PyTorch to easily write and train Graph Neural Networks (GNNs) for a wide range of applications related to structured data.\n", + "\n", + "For protein classification task, we used it in an unsupervised manner, following [PyG's unsupervised PPI example](https://github.com/pyg-team/pytorch_geometric/blob/master/examples/graph_sage_unsup_ppi.py).\n", + "For financial transaction classification task, we used it in a supervised manner, directly using the node labels with supervised classification loss.\n", + "\n", + "Since the inductive learning mode is being used, the locally learnt model (a representation encoding / classification network) is irrelevant to the candidate graph, we are able to use the basic [FedAvg](https://arxiv.org/abs/1602.05629) as the federated learning algorithm. The workflow is Scatter and Gather (SAG).\n", + "\n", + "\n", + "Below we listed steps to run this example." + ] + }, + { + "cell_type": "markdown", + "id": "a5a0292c-78b6-4bde-96d6-699dae996173", + "metadata": {}, + "source": [ + "## 1. Setup NVFLARE\n", + "\n", + "Follow the [Getting_Started](https://nvflare.readthedocs.io/en/main/getting_started.html) to setup virtual environment and install NVFLARE\n", + "\n", + "We also provide a [Notebook](../../nvflare_setup.ipynb) for this setup process. \n", + "\n", + "Assume you have already setup the venv, lets first install required packages." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f4130b15-09e6-456f-a3c7-87c8ee9e07f0", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "%pip install -r requirements.txt" + ] + }, + { + "cell_type": "markdown", + "id": "1d872d8a-9e44-49dd-94b1-7862b3815ffe", + "metadata": {}, + "source": [ + "To support functions of PyTorch Geometric necessary for this example, we need extra dependencies. Please refer to [installation guide](https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html) and install accordingly:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f906a1c9-dce0-476c-be65-79ebd8ad5da9", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "%pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.1.0+cpu.html" + ] + }, + { + "cell_type": "markdown", + "id": "e93b1bf2-6157-4ab6-9766-819450304038", + "metadata": {}, + "source": [ + "## 2. Data preparation \n", + "This example uses two datasets: \n", + "- For Protein Classification, the PPI dataset is available from torch_geometric's dataset API. \n", + "- For Financial Transaction Classification, we first download the [Elliptic++](https://github.com/git-disl/EllipticPlusPlus) dataset to `/tmp/nvflare/datasets/elliptic_pp` folder. In this example, we will use the following three files:\n", + " - `txs_classes.csv`: transaction id and its class (licit or illicit)\n", + " - `txs_edgelist.csv`: connections for transaction ids \n", + " - `txs_features.csv`: transaction id and its features" + ] + }, + { + "cell_type": "markdown", + "id": "af257e69-2bb7-49b6-ac6c-f007b0e6618e", + "metadata": {}, + "source": [ + "## 3. Local Experiments\n", + "For comparison with federated learning results, we first perform local experiments on each client's data and the whole dataset. Here we simulate 2 clients with uniform data split (client_id = 0 means the whole dataset). The 6 experiments will take a while to finish. The default epoch number is set to 70. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fdb7290a-48ff-4e80-be58-5e6b0e0f9379", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [], + "source": [ + "! python3 code/graphsage_protein_local.py --client_id 0\n", + "! python3 code/graphsage_protein_local.py --client_id 1\n", + "! python3 code/graphsage_protein_local.py --client_id 2 " + ] + }, + { + "cell_type": "markdown", + "id": "9a2d55cf-4f7a-4030-8cba-b1619fdf1614", + "metadata": {}, + "source": [ + "And for finance experiment" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f4cf2c09-1f78-4d28-9b86-af9f9cf86479", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [], + "source": [ + "! python3 code/graphsage_finance_local.py --client_id 0\n", + "! python3 code/graphsage_finance_local.py --client_id 1\n", + "! python3 code/graphsage_finance_local.py --client_id 2 " + ] + }, + { + "cell_type": "markdown", + "id": "d178c6dc-c180-4ca6-8dea-3b0fe147665b", + "metadata": {}, + "source": [ + "## 4. Prepare NVFlare job based on GNN template\n", + "We are using NVFlare's FL simulator to run the FL experiments. First, we create jobs using GNN template. We reuse the job templates from [sag_gnn](../../../job_templates/sag_gnn), let's set the job template path with the following command." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fd885e6b-ae4d-40aa-b89d-fe34217ad3da", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [], + "source": [ + "! nvflare config -jt ../../../job_templates/" + ] + }, + { + "cell_type": "markdown", + "id": "f608a992-5096-4452-8775-b89987970a75", + "metadata": {}, + "source": [ + "Then we can check the available templates with the following command." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1f8041c7-7fae-4c8a-8e07-1c6a6d59e541", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [], + "source": [ + "! nvflare job list_templates" + ] + }, + { + "cell_type": "markdown", + "id": "5bad4f55-d582-4f37-a523-927dc015e564", + "metadata": {}, + "source": [ + "We shall see `sag_gnn` from the above command. We then create jobs using this template, and set local epochs to 10 with 7 rounds of FL to match local experiments' 70 epoch default training." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7971a800-70fc-4213-96ed-c157801b5a11", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [], + "source": [ + "! nvflare job create -force -j \"/tmp/nvflare/jobs/gnn_protein\" -w \"sag_gnn\" -sd \"code\" \\\n", + " -f app_1/config_fed_client.conf app_script=\"graphsage_protein_fl.py\" app_config=\"--client_id 1 --epochs 10\" \\\n", + " -f app_2/config_fed_client.conf app_script=\"graphsage_protein_fl.py\" app_config=\"--client_id 2 --epochs 10\" \\\n", + " -f app_server/config_fed_server.conf num_rounds=7 key_metric=\"validation_f1\" model_class_path=\"torch_geometric.nn.GraphSAGE\" components[0].args.model.args.in_channels=50 components[0].args.model.args.hidden_channels=64 components[0].args.model.args.num_layers=2 components[0].args.model.args.out_channels=64 " + ] + }, + { + "cell_type": "markdown", + "id": "675bff95-dcfa-4a47-9a05-460da16760ef", + "metadata": {}, + "source": [ + "And for finance experiment" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a6d0b643-31f0-4d52-ae3c-1fafcd404072", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [], + "source": [ + "! nvflare job create -force -j \"/tmp/nvflare/jobs/gnn_finance\" -w \"sag_gnn\" -sd \"code\" \\\n", + " -f app_1/config_fed_client.conf app_script=\"graphsage_finance_fl.py\" app_config=\"--client_id 1 --epochs 10\" \\\n", + " -f app_2/config_fed_client.conf app_script=\"graphsage_finance_fl.py\" app_config=\"--client_id 2 --epochs 10\" \\\n", + " -f app_server/config_fed_server.conf num_rounds=7 key_metric=\"validation_auc\" model_class_path=\"pyg_sage.SAGE\" components[0].args.model.args.in_channels=165 components[0].args.model.args.hidden_channels=256 components[0].args.model.args.num_layers=3 components[0].args.model.args.num_classes=2 \n" + ] + }, + { + "cell_type": "markdown", + "id": "bd0713e2-e393-41c0-9da0-392535cf8a54", + "metadata": {}, + "source": [ + "## 5. Run simulated kmeans experiment\n", + "Now that we have the jobs ready, we run the experiment using Simulator." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9bb6cab4-9c24-400a-bc3c-f1e4a6d5a346", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "! nvflare simulator -w /tmp/nvflare/gnn/protein_fl_workspace -n 2 -t 2 /tmp/nvflare/jobs/gnn_protein" + ] + }, + { + "cell_type": "markdown", + "id": "98c64648-1d09-42da-bd48-9a6ac48587af", + "metadata": {}, + "source": [ + "And for finance experiment" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a9f256a0-ae99-4a7e-8bc2-e7fc8de2e6f6", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "! nvflare simulator -w /tmp/nvflare/gnn/finance_fl_workspace -n 2 -t 2 /tmp/nvflare/jobs/gnn_finance" + ] + }, + { + "cell_type": "markdown", + "id": "913e9ee2-e993-442d-a525-d2baf92af539", + "metadata": {}, + "source": [ + "## 6. Result visualization\n", + "Results from both local and federated experiments can be visualized in tensorboard." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a6814434-4e6d-4460-b480-709cb3e77cc8", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "%load_ext tensorboard\n", + "%tensorboard --logdir /tmp/nvflare/gnn" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0f6ae6cb-12df-4279-b6af-9c4d356e727e", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "nvflare_example", + "language": "python", + "name": "nvflare_example" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.16" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/advanced/gnn/requirements.txt b/examples/advanced/gnn/requirements.txt index 129158b55d..c469afd535 100644 --- a/examples/advanced/gnn/requirements.txt +++ b/examples/advanced/gnn/requirements.txt @@ -1,4 +1,3 @@ -nvflare torch torch_geometric tensorboard From 73b0bdcab27f5bb7ae425a38a464638355113828 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Wed, 17 Jan 2024 19:25:14 -0800 Subject: [PATCH 07/39] Fix FLModelUtil (#2291) --- nvflare/app_common/utils/fl_model_utils.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/nvflare/app_common/utils/fl_model_utils.py b/nvflare/app_common/utils/fl_model_utils.py index 204928fd13..2d84daa14f 100644 --- a/nvflare/app_common/utils/fl_model_utils.py +++ b/nvflare/app_common/utils/fl_model_utils.py @@ -97,7 +97,11 @@ def from_shareable(shareable: Shareable, fl_ctx: Optional[FLContext] = None) -> params = None meta = {} - try: + submit_model_name = shareable.get_header(AppConstants.SUBMIT_MODEL_NAME) + if submit_model_name: + # this only happens in cross-site eval right now + meta[MetaKey.SUBMIT_MODEL_NAME] = submit_model_name + else: dxo = from_shareable(shareable) meta = dict(dxo.meta) if dxo.data_kind == DataKind.METRICS: @@ -115,10 +119,6 @@ def from_shareable(shareable: Shareable, fl_ctx: Optional[FLContext] = None) -> if MetaKey.INITIAL_METRICS in meta: metrics = meta[MetaKey.INITIAL_METRICS] - except: - # this only happens in cross-site eval right now - submit_model_name = shareable.get_header(AppConstants.SUBMIT_MODEL_NAME) - meta[MetaKey.SUBMIT_MODEL_NAME] = submit_model_name current_round = shareable.get_header(AppConstants.CURRENT_ROUND, None) total_rounds = shareable.get_header(AppConstants.NUM_ROUNDS, None) From af027a03604c9160fd71dd10c4b7cbceb738f386 Mon Sep 17 00:00:00 2001 From: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Date: Fri, 19 Jan 2024 17:09:02 -0800 Subject: [PATCH 08/39] Check invalid input directory in nvflare config (#2295) * check invalid input directory * check invalid input directory --- nvflare/cli.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/nvflare/cli.py b/nvflare/cli.py index 316973f8ba..eee979f7a7 100644 --- a/nvflare/cli.py +++ b/nvflare/cli.py @@ -116,6 +116,9 @@ def def_config_parser(sub_cmd): def handle_config_cmd(args): config_file_path, nvflare_config = get_hidden_config() + if not args.job_templates_dir or not os.path.isdir(args.job_templates_dir): + raise ValueError(f"job_templates_dir='{args.job_templates_dir}', it is not a directory") + nvflare_config = create_startup_kit_config(nvflare_config, args.startup_kit_dir) nvflare_config = create_poc_workspace_config(nvflare_config, args.poc_workspace_dir) nvflare_config = create_job_template_config(nvflare_config, args.job_templates_dir) From a0e8523f89d9ee1fceb471f57e924e48c172e326 Mon Sep 17 00:00:00 2001 From: Isaac Yang Date: Thu, 11 Jan 2024 11:38:21 -0800 Subject: [PATCH 09/39] AWS server and client OK Azure template updated Refactor sub_start.sh on both server and client Address PR comments Address a few PR comments --- nvflare/dashboard/application/blob.py | 215 ++--- nvflare/dashboard/cli.py | 3 +- nvflare/lighter/dummy_project.yml | 5 +- nvflare/lighter/ha_project.yml | 5 +- nvflare/lighter/impl/aws_template.yml | 261 ++++++ nvflare/lighter/impl/azure_template.yml | 517 +++++++++++ nvflare/lighter/impl/master_template.yml | 1073 +--------------------- nvflare/lighter/impl/static_file.py | 104 +-- nvflare/lighter/impl/template.py | 6 +- nvflare/lighter/impl/workspace.py | 17 +- nvflare/lighter/tplt_utils.py | 72 ++ nvflare/lighter/utils.py | 72 ++ 12 files changed, 1044 insertions(+), 1306 deletions(-) create mode 100644 nvflare/lighter/impl/aws_template.yml create mode 100644 nvflare/lighter/impl/azure_template.yml diff --git a/nvflare/dashboard/application/blob.py b/nvflare/dashboard/application/blob.py index 389b5bb4c6..174e2135bb 100644 --- a/nvflare/dashboard/application/blob.py +++ b/nvflare/dashboard/application/blob.py @@ -25,24 +25,17 @@ lighter_folder = os.path.dirname(utils.__file__) template = utils.load_yaml(os.path.join(lighter_folder, "impl", "master_template.yml")) - - -def get_csp_template(csp, participant, template): - return template[f"{csp}_start_{participant}_sh"] +supported_csps = ["aws", "azure"] +for csp in supported_csps: + csp_template_file = os.path.join(lighter_folder, "impl", f"{csp}_template.yml") + if os.path.exists(csp_template_file): + template.update(utils.load_yaml(csp_template_file)) def get_csp_start_script_name(csp): return f"{csp}_start.sh" -def _write(file_full_path, content, mode, exe=False): - mode = mode + "w" - with open(file_full_path, mode) as f: - f.write(content) - if exe: - os.chmod(file_full_path, 0o755) - - def gen_overseer(key): project = Project.query.first() entity = Entity(project.overseer) @@ -54,21 +47,19 @@ def gen_overseer(key): dest_dir = os.path.join(overseer_dir, "startup") os.mkdir(overseer_dir) os.mkdir(dest_dir) - _write( + utils._write( os.path.join(dest_dir, "start.sh"), template["start_ovsr_sh"], "t", exe=True, ) - _write( + utils._write( os.path.join(dest_dir, "gunicorn.conf.py"), utils.sh_replace(template["gunicorn_conf_py"], {"port": "8443"}), "t", exe=False, ) - _write(os.path.join(dest_dir, "overseer.crt"), cert_pair.ser_cert, "b", exe=False) - _write(os.path.join(dest_dir, "overseer.key"), cert_pair.ser_pri_key, "b", exe=False) - _write(os.path.join(dest_dir, "rootCA.pem"), project.root_cert, "b", exe=False) + utils._write_pki(type="overseer", dest_dir=dest_dir, cert_pair=cert_pair, root_cert=project.root_cert) run_args = ["zip", "-rq", "-P", key, "tmp.zip", "."] subprocess.run(run_args, cwd=tmp_dir) fileobj = io.BytesIO() @@ -121,6 +112,8 @@ def gen_server(key, first_server=True): "ha_mode": "true" if project.ha_mode else "false", "docker_image": project.app_location.split(" ")[-1] if project.app_location else "nvflare/nvflare", "org_name": "", + "type": "server", + "cln_uid": "", } tplt = tplt_utils.Template(template) with tempfile.TemporaryDirectory() as tmp_dir: @@ -128,82 +121,33 @@ def gen_server(key, first_server=True): dest_dir = os.path.join(server_dir, "startup") os.mkdir(server_dir) os.mkdir(dest_dir) - _write(os.path.join(dest_dir, "fed_server.json"), json.dumps(config, indent=2), "t") - _write( - os.path.join(dest_dir, "docker.sh"), - utils.sh_replace(template["docker_svr_sh"], replacement_dict), - "t", - exe=True, - ) - _write( - os.path.join(dest_dir, "start.sh"), - utils.sh_replace(template["start_svr_sh"], replacement_dict), - "t", - exe=True, - ) - _write( - os.path.join(dest_dir, "sub_start.sh"), - utils.sh_replace(template["sub_start_svr_sh"], replacement_dict), - "t", - exe=True, - ) - _write( - os.path.join(dest_dir, "stop_fl.sh"), - template["stop_fl_sh"], - "t", - exe=True, + utils._write_common( + type="server", + dest_dir=dest_dir, + template=template, + tplt=tplt, + replacement_dict=replacement_dict, + config=config, ) - _write(os.path.join(dest_dir, "server.crt"), cert_pair.ser_cert, "b", exe=False) - _write(os.path.join(dest_dir, "server.key"), cert_pair.ser_pri_key, "b", exe=False) - _write(os.path.join(dest_dir, "rootCA.pem"), project.root_cert, "b", exe=False) + utils._write_pki(type="server", dest_dir=dest_dir, cert_pair=cert_pair, root_cert=project.root_cert) if not project.ha_mode: - _write( - os.path.join(dest_dir, get_csp_start_script_name("azure")), - utils.sh_replace( - tplt.get_cloud_script_header() + get_csp_template("azure", "svr", template), - {"server_name": entity.name, "ORG": ""}, - ), - "t", - exe=True, - ) - _write( - os.path.join(dest_dir, get_csp_start_script_name("aws")), - utils.sh_replace( - tplt.get_cloud_script_header() + get_csp_template("aws", "svr", template), - {"server_name": entity.name, "ORG": ""}, - ), - "t", - exe=True, - ) + for csp in supported_csps: + utils._write( + os.path.join(dest_dir, get_csp_start_script_name(csp)), + tplt.get_start_sh(csp=csp, type="server", entity=entity), + "t", + exe=True, + ) signatures = utils.sign_all(dest_dir, deserialize_ca_key(project.root_key)) json.dump(signatures, open(os.path.join(dest_dir, "signature.json"), "wt")) # local folder creation dest_dir = os.path.join(server_dir, "local") os.mkdir(dest_dir) - _write( - os.path.join(dest_dir, "log.config.default"), - template["log_config"], - "t", - ) - _write( - os.path.join(dest_dir, "resources.json.default"), - template["local_server_resources"], - "t", - ) - _write( - os.path.join(dest_dir, "privacy.json.sample"), - template["sample_privacy"], - "t", - ) - _write( - os.path.join(dest_dir, "authorization.json.default"), - template["default_authz"], - "t", - ) + utils._write_local(type="server", dest_dir=dest_dir, template=template) # workspace folder file - _write( + utils._write( os.path.join(server_dir, "readme.txt"), template["readme_fs"], "t", @@ -233,6 +177,8 @@ def gen_client(key, id): "config_folder": "config", "docker_image": project.app_location.split(" ")[-1] if project.app_location else "nvflare/nvflare", "org_name": entity.org, + "type": "client", + "cln_uid": f"uid={entity.name}", } if project.ha_mode: overseer_agent = {"path": "nvflare.ha.overseer_agent.HttpOverseerAgent"} @@ -254,85 +200,34 @@ def gen_client(key, id): os.mkdir(client_dir) os.mkdir(dest_dir) - _write(os.path.join(dest_dir, "fed_client.json"), json.dumps(config, indent=2), "t") - _write( - os.path.join(dest_dir, "docker.sh"), - utils.sh_replace(template["docker_cln_sh"], replacement_dict), - "t", - exe=True, - ) - _write( - os.path.join(dest_dir, "start.sh"), - template["start_cln_sh"], - "t", - exe=True, - ) - _write( - os.path.join(dest_dir, "sub_start.sh"), - utils.sh_replace(template["sub_start_cln_sh"], replacement_dict), - "t", - exe=True, - ) - _write( - os.path.join(dest_dir, "stop_fl.sh"), - template["stop_fl_sh"], - "t", - exe=True, - ) - _write(os.path.join(dest_dir, "client.crt"), cert_pair.ser_cert, "b", exe=False) - _write(os.path.join(dest_dir, "client.key"), cert_pair.ser_pri_key, "b", exe=False) - _write(os.path.join(dest_dir, "rootCA.pem"), project.root_cert, "b", exe=False) - _write( - os.path.join(dest_dir, get_csp_start_script_name("azure")), - utils.sh_replace( - tplt.get_cloud_script_header() + get_csp_template("azure", "cln", template), - {"SITE": entity.name, "ORG": entity.org}, - ), - "t", - exe=True, - ) - _write( - os.path.join(dest_dir, get_csp_start_script_name("aws")), - utils.sh_replace( - tplt.get_cloud_script_header() + get_csp_template("aws", "cln", template), - {"SITE": entity.name, "ORG": entity.org}, - ), - "t", - exe=True, + utils._write_pki(type="client", dest_dir=dest_dir, cert_pair=cert_pair, root_cert=project.root_cert) + utils._write_common( + type="client", + dest_dir=dest_dir, + template=template, + tplt=tplt, + replacement_dict=replacement_dict, + config=config, ) + + for csp in supported_csps: + utils._write( + os.path.join(dest_dir, get_csp_start_script_name(csp)), + tplt.get_start_sh(csp=csp, type="client", entity=entity), + "t", + exe=True, + ) + signatures = utils.sign_all(dest_dir, deserialize_ca_key(project.root_key)) json.dump(signatures, open(os.path.join(dest_dir, "signature.json"), "wt")) # local folder creation dest_dir = os.path.join(client_dir, "local") os.mkdir(dest_dir) - _write( - os.path.join(dest_dir, "log.config.default"), - template["log_config"], - "t", - ) - resources = json.loads(template["local_client_resources"]) - for component in resources["components"]: - if "nvflare.app_common.resource_managers.gpu_resource_manager.GPUResourceManager" == component["path"]: - component["args"] = json.loads(client.capacity.capacity) - break - _write( - os.path.join(dest_dir, "resources.json.default"), - json.dumps(resources, indent=2), - "t", - ) - _write( - os.path.join(dest_dir, "privacy.json.sample"), - template["sample_privacy"], - "t", - ) - _write( - os.path.join(dest_dir, "authorization.json.default"), - template["default_authz"], - "t", - ) + utils._write_local(type="client", dest_dir=dest_dir, template=template, capacity=client.capacity.capacity) + # workspace folder file - _write( + utils._write( os.path.join(client_dir, "readme.txt"), template["readme_fc"], "t", @@ -378,16 +273,14 @@ def gen_user(key, id): os.mkdir(user_dir) os.mkdir(dest_dir) - _write(os.path.join(dest_dir, "fed_admin.json"), json.dumps(config, indent=2), "t") - _write( + utils._write(os.path.join(dest_dir, "fed_admin.json"), json.dumps(config, indent=2), "t") + utils._write( os.path.join(dest_dir, "fl_admin.sh"), utils.sh_replace(template["fl_admin_sh"], replacement_dict), "t", exe=True, ) - _write(os.path.join(dest_dir, "client.crt"), cert_pair.ser_cert, "b", exe=False) - _write(os.path.join(dest_dir, "client.key"), cert_pair.ser_pri_key, "b", exe=False) - _write(os.path.join(dest_dir, "rootCA.pem"), project.root_cert, "b", exe=False) + utils._write_pki(type="client", dest_dir=dest_dir, cert_pair=cert_pair, root_cert=project.root_cert) signatures = utils.sign_all(dest_dir, deserialize_ca_key(project.root_key)) json.dump(signatures, open(os.path.join(dest_dir, "signature.json"), "wt")) @@ -396,12 +289,12 @@ def gen_user(key, id): os.mkdir(dest_dir) # workspace folder file - _write( + utils._write( os.path.join(user_dir, "readme.txt"), template["readme_am"], "t", ) - _write( + utils._write( os.path.join(user_dir, "system_info.ipynb"), utils.sh_replace(template["adm_notebook"], replacement_dict), "t", diff --git a/nvflare/dashboard/cli.py b/nvflare/dashboard/cli.py index a31409b545..58e45dbe10 100644 --- a/nvflare/dashboard/cli.py +++ b/nvflare/dashboard/cli.py @@ -21,7 +21,6 @@ import docker import nvflare from nvflare.apis.utils.format_check import name_check -from nvflare.dashboard.application.blob import _write from nvflare.lighter import tplt_utils, utils supported_csp = ("azure", "aws") @@ -146,7 +145,7 @@ def cloud(args): dsb_start = template[f"{csp}_start_dsb_sh"] version = nvflare.__version__ replacement_dict = {"NVFLARE": f"nvflare=={version}", "START_OPT": f"-i {args.image}" if args.image else ""} - _write( + utils._write( dest, utils.sh_replace(tplt.get_cloud_script_header() + dsb_start, replacement_dict), "t", diff --git a/nvflare/lighter/dummy_project.yml b/nvflare/lighter/dummy_project.yml index 51d8cc6379..fb5a759b95 100644 --- a/nvflare/lighter/dummy_project.yml +++ b/nvflare/lighter/dummy_project.yml @@ -28,7 +28,10 @@ participants: builders: - path: nvflare.lighter.impl.workspace.WorkspaceBuilder args: - template_file: master_template.yml + template_file: + - master_template.yml + - aws_template.yml + - azure_template.yml - path: nvflare.lighter.impl.template.TemplateBuilder - path: nvflare.lighter.impl.static_file.StaticFileBuilder args: diff --git a/nvflare/lighter/ha_project.yml b/nvflare/lighter/ha_project.yml index 2a5fecad28..7216dcc762 100644 --- a/nvflare/lighter/ha_project.yml +++ b/nvflare/lighter/ha_project.yml @@ -40,7 +40,10 @@ participants: builders: - path: nvflare.lighter.impl.workspace.WorkspaceBuilder args: - template_file: master_template.yml + template_file: + - master_template.yml + - aws_template.yml + - azure_template.yml - path: nvflare.lighter.impl.template.TemplateBuilder - path: nvflare.lighter.impl.docker.DockerBuilder args: diff --git a/nvflare/lighter/impl/aws_template.yml b/nvflare/lighter/impl/aws_template.yml new file mode 100644 index 0000000000..8ba14d6f2d --- /dev/null +++ b/nvflare/lighter/impl/aws_template.yml @@ -0,0 +1,261 @@ +aws_start_sh: | + VM_NAME=nvflare_{~~type~~} + SECURITY_GROUP=nvflare_{~~type~~}_sg_$RANDOM + DEST_FOLDER=/var/tmp/cloud + KEY_PAIR=NVFlare{~~type~~}KeyPair + KEY_FILE=${KEY_PAIR}.pem + + echo "This script requires aws (AWS CLI), sshpass, dig and jq. Now checking if they are installed." + + check_binary aws "Please see https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html on how to install it on your system." + check_binary sshpass "Please install it first." + check_binary dig "Please install it first." + check_binary jq "Please install it first." + + if [ -z ${image_name+x} ] + then + container=false + else + container=true + fi + + if [ $container = true ] + then + AMI_IMAGE=ami-06b8d5099f3a8d79d + EC2_TYPE=t2.xlarge + REGION=us-west-2 + else + AMI_IMAGE=ami-04bad3c587fe60d89 + EC2_TYPE=t2.small + REGION=us-west-2 + fi + + if [ -z ${config_file+x} ] + then + useDefault=true + else + useDefault=false + . $config_file + report_status "$?" "Loading config file" + fi + + + if [ $useDefault = true ] + then + while true + do + prompt AMI_IMAGE "Cloud AMI image, press ENTER to accept default ${AMI_IMAGE}: " + prompt EC2_TYPE "Cloud EC2 type, press ENTER to accept default ${EC2_TYPE}: " + prompt REGIION "Cloud EC2 region, press ENTER to accept default ${REGION}: " + prompt ans "region = ${REGION}, ami image = ${AMI_IMAGE}, EC2 type = ${EC2_TYPE}, OK? (Y/n) " + if [[ $ans = "" ]] || [[ $ans =~ ^(y|Y)$ ]] + then + break + fi + done + fi + + if [ $container = false ] + then + echo "If the {~~type~~} requires additional dependencies, please copy the requirements.txt to ${DIR}." + prompt ans "Press ENTER when it's done or no additional dependencies. " + fi + + cd $DIR/.. + # Generate key pair + + echo "Generating key pair for VM" + + aws ec2 delete-key-pair --key-name $KEY_PAIR > /dev/null 2>&1 + rm -rf $KEY_FILE + aws ec2 create-key-pair --key-name $KEY_PAIR --query 'KeyMaterial' --output text > $KEY_FILE + report_status "$?" "creating key pair" + chmod 400 $KEY_FILE + + # Generate Security Group + # Try not reusing existing security group because we have to modify it for our own need. + sg_id=$(aws ec2 create-security-group --group-name $SECURITY_GROUP --description "NVFlare security group" | jq -r .GroupId) + report_status "$?" "creating security group" + my_public_ip=$(dig +short myip.opendns.com @resolver1.opendns.com) + if [ "$?" -eq 0 ] && [[ "$my_public_ip" =~ ^(([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))\.){3}([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))$ ]] + then + aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr ${my_public_ip}/32 > /tmp/sec_grp.log + else + echo "getting my public IP failed, please manually configure the inbound rule to limit SSH access" + aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr 0.0.0.0/0 > /tmp/sec_grp.log + fi + {~~inbound_rule~~} + report_status "$?" "creating security group rules" + + # Start provisioning + + echo "Creating VM at region $REGION, may take a few minutes." + + aws ec2 run-instances --region $REGION --image-id $AMI_IMAGE --count 1 --instance-type $EC2_TYPE --key-name $KEY_PAIR --security-group-ids $sg_id > vm_create.json + report_status "$?" "creating VM" + instance_id=$(jq -r .Instances[0].InstanceId vm_create.json) + + aws ec2 wait instance-status-ok --instance-ids $instance_id + aws ec2 describe-instances --instance-ids $instance_id > vm_result.json + + IP_ADDRESS=$(jq -r .Reservations[0].Instances[0].PublicIpAddress vm_result.json) + + echo "VM created with IP address: ${IP_ADDRESS}" + + echo "Copying files to $VM_NAME" + DEST_SITE=ubuntu@${IP_ADDRESS} + DEST=${DEST_SITE}:${DEST_FOLDER} + echo "Destination folder is ${DEST}" + scp -q -i $KEY_FILE -r -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $PWD $DEST + report_status "$?" "copying startup kits to VM" + + if [ $container = true ] + then + echo "Launching container with docker option ${DOCKER_OPTION}." + ssh -f -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ + "docker run -d -v ${DEST_FOLDER}:${DEST_FOLDER} --network host ${DOCKER_OPTION} ${image_name} \ + /bin/bash -c \"python -u -m nvflare.private.fed.app.{~~type~~}.{~~type~~}_train -m ${DEST_FOLDER} \ + -s fed_{~~type~~}.json --set {~~cln_uid~~} secure_train=true config_folder=config org={~~ORG~~} \" " > /tmp/nvflare.log 2>&1 + report_status "$?" "launching container" + else + echo "Installing packages in $VM_NAME, may take a few minutes." + ssh -f -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ + "pwd && wget -q https://bootstrap.pypa.io/get-pip.py && \ + python3 get-pip.py && python3 -m pip install nvflare && \ + touch ${DEST_FOLDER}/startup/requirements.txt && \ + python3 -m pip install -r ${DEST_FOLDER}/startup/requirements.txt && \ + nohup ${DEST_FOLDER}/startup/start.sh && sleep 20 && \ + exit" > /tmp/nvflare.log 2>&1 + report_status "$?" "installing packages" + fi + + echo "System was provisioned" + echo "To terminate the EC2 instance, run the following command." + echo "aws ec2 terminate-instances --instance-ids ${instance_id}" + echo "Other resources provisioned" + echo "security group: ${SECURITY_GROUP}" + echo "key pair: ${KEY_PAIR}" + +aws_start_dsb_sh: | + VM_NAME=nvflare_dashboard + AMI_IMAGE=ami-04bad3c587fe60d89 + EC2_TYPE=t2.small + SECURITY_GROUP=nvflare_dashboard_sg_$RANDOM + REGION=us-west-2 + ADMIN_USERNAME=ubuntu + DEST_FOLDER=/home/${ADMIN_USERNAME} + KEY_PAIR=NVFlareDashboardKeyPair + KEY_FILE=${KEY_PAIR}.pem + + echo "This script requires aws (AWS CLI), sshpass, dig and jq. Now checking if they are installed." + + check_binary aws "Please see https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html on how to install it on your system." + check_binary sshpass "Please install it first." + check_binary dig "Please install it first." + check_binary jq "Please install it first." + + echo "One initial user will be created when starting dashboard." + echo "Please enter the email address for this user." + read email + credential="${email}:$RANDOM" + + # Generate key pair + + echo "Generating key pair for VM" + + aws ec2 delete-key-pair --key-name $KEY_PAIR > /dev/null 2>&1 + rm -rf $KEY_FILE + aws ec2 create-key-pair --key-name $KEY_PAIR --query 'KeyMaterial' --output text > $KEY_FILE + report_status "$?" "creating key pair" + chmod 400 $KEY_FILE + + # Generate Security Group + + sg_id=$(aws ec2 create-security-group --group-name $SECURITY_GROUP --description "NVFlare security group" | jq -r .GroupId) + report_status "$?" "creating security group" + echo "Security group id: ${sg_id}" + my_public_ip=$(dig +short myip.opendns.com @resolver1.opendns.com) + if [ "$?" -eq 0 ] && [[ "$my_public_ip" =~ ^(([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))\.){3}([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))$ ]] + then + aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr ${my_public_ip}/32 > /tmp/sec_grp.log + else + echo "getting my public IP failed, please manually configure the inbound rule to limit SSH access" + aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr 0.0.0.0/0 > /tmp/sec_grp.log + fi + aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 443 --cidr 0.0.0.0/0 >> /tmp/sec_grp.log + report_status "$?" "creating security group rules" + + # Start provisioning + + echo "Creating VM at region $REGION, may take a few minutes." + + aws ec2 run-instances --region $REGION --image-id $AMI_IMAGE --count 1 --instance-type $EC2_TYPE --key-name $KEY_PAIR --security-group-ids $sg_id > vm_create.json + report_status "$?" "creating VM" + instance_id=$(jq -r .Instances[0].InstanceId vm_create.json) + + aws ec2 wait instance-status-ok --instance-ids $instance_id + aws ec2 describe-instances --instance-ids $instance_id > vm_result.json + + IP_ADDRESS=$(jq -r .Reservations[0].Instances[0].PublicIpAddress vm_result.json) + + echo "VM created with IP address: ${IP_ADDRESS}" + + echo "Installing docker engine in $VM_NAME, may take a few minutes." + DEST_SITE=${ADMIN_USERNAME}@${IP_ADDRESS} + scripts=$(cat << 'EOF' + sudo apt-get update && \ + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y ca-certificates curl gnupg lsb-release && \ + sudo mkdir -p /etc/apt/keyrings && \ + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \ + echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \ + sudo apt-get update && \ + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y docker-ce docker-ce-cli containerd.io + EOF + ) + ssh -t -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} "$scripts" > /tmp/docker_engine.log + report_status "$?" "installing docker engine" + ssh -t -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} "sudo usermod -aG docker $ADMIN_USERNAME && exit" >> /tmp/docker_engine.log + report_status "$?" "installing docker engine" + + echo "Installing nvflare in $VM_NAME, may take a few minutes." + ssh -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ + "export PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin && \ + wget -q https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && \ + python3 -m pip install {~~NVFLARE~~} && \ + mkdir -p ./cert && \ + exit" > /tmp/nvflare.json + report_status "$?" "installing nvflare" + + echo "Checking if certificate (web.crt) and private key (web.key) are available" + if [[ -f "web.crt" && -f "web.key" ]]; then + CERT_FOLDER=${DEST_SITE}:${DEST_FOLDER}/cert + echo "Cert folder is ${CERT_FOLDER}" + scp -i $KEY_FILE -r -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null web.{crt,key} $CERT_FOLDER + report_status "$?" "copying cert/key to VM ${CERT_FOLDER} folder" + secure=true + else + echo "No web.crt and web.key found" + secure=false + fi + + echo "Starting dashboard" + ssh -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ + "export PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin && \ + python3 -m nvflare.dashboard.cli --start -f ${DEST_FOLDER} --cred ${credential} {~~START_OPT~~}" > /tmp/dashboard.json + + echo "Dashboard url is running at IP address ${IP_ADDRESS}, listening to port 443." + if [ "$secure" = true ] + then + echo "URL is https://${IP_ADDRESS}" + else + echo "URL is http://${IP_ADDRESS}:443" + fi + echo "Note: you may need to configure DNS server with your DNS hostname and the above IP address." + echo "Project admin credential (username:password) is ${credential} ." + echo "To terminate the EC2 instance, run the following command." + echo "aws ec2 terminate-instances --instance-ids ${instance_id}" + echo "Other resources provisioned" + echo "security group: ${SECURITY_GROUP}" + echo "key pair: ${KEY_PAIR}" diff --git a/nvflare/lighter/impl/azure_template.yml b/nvflare/lighter/impl/azure_template.yml new file mode 100644 index 0000000000..9c42a10cf3 --- /dev/null +++ b/nvflare/lighter/impl/azure_template.yml @@ -0,0 +1,517 @@ +azure_start_svr_header_sh: | + RESOURCE_GROUP=nvflare_rg + VM_NAME=nvflare_server + VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest + VM_SIZE=Standard_B2ms + NSG_NAME=nvflare_nsgs + ADMIN_USERNAME=nvflare + PASSWORD="NVFl@r3_P@88"$RANDOM"w0rd" + DEST_FOLDER=/var/tmp/cloud + NIC_NAME=${VM_NAME}VMNic + SERVER_NAME={~~server_name~~} + FL_PORT=8002 + ADMIN_PORT=8003 + + echo "This script requires az (Azure CLI), sshpass and jq. Now checking if they are installed." + + check_binary az "Please see https://learn.microsoft.com/en-us/cli/azure/install-azure-cli on how to install it on your system." + check_binary sshpass "Please install it first." + check_binary jq "Please install it first." + + self_dns=true + if [[ "$SERVER_NAME" = *".cloudapp.azure.com"* ]] + then + DNS_TAG=$(echo $SERVER_NAME | cut -d "." -f 1) + DERIVED_LOCATION=$(echo $SERVER_NAME | cut -d "." -f 2) + LOCATION=$DERIVED_LOCATION + self_dns=false + else + echo "Warning: ${SERVER_NAME} does not end with .cloudapp.azure.com." + echo "The cloud launch process will not create the domain name for you." + echo "Please use your own DNS to set the information." + LOCATION=westus2 + fi + + if [ -z ${image_name+x} ] + then + container=false + else + container=true + fi + + if [ $container = true ] + then + VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest + VM_SIZE=Standard_D8s_v3 + else + VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest + VM_SIZE=Standard_B2ms + fi + + if [ -z ${config_file+x} ] + then + useDefault=true + else + useDefault=false + . $config_file + report_status "$?" "Loading config file" + if [ $self_dns = false ] && [ $DERIVED_LOCATION != $LOCATION ] + then + echo "Server name implies LOCATION=${DERIVED_LOCATION} but the config file specifies LOCATION=${LOCATION}. Unable to continue provisioning." + exit 1 + fi + fi + + if [ $useDefault = true ] + then + while true + do + prompt VM_IMAGE "Cloud VM image, press ENTER to accept default ${VM_IMAGE}: " + prompt VM_SIZE "Cloud VM size, press ENTER to accept default ${VM_SIZE}: " + if [ $self_dns = true ] + then + prompt LOCATION "Cloud location, press ENTER to accept default ${LOCATION}: " + prompt ans "VM image = ${VM_IMAGE}, VM size = ${VM_SIZE}, location = ${LOCATION}, OK? (Y/n) " + else + prompt ans "VM image = ${VM_IMAGE}, VM size = ${VM_SIZE}, OK? (Y/n) " + fi + if [[ $ans = "" ]] || [[ $ans =~ ^(y|Y)$ ]]; then break; fi + done + fi + + if [ $container = false ] + then + echo "If the client requires additional dependencies, please copy the requirements.txt to ${DIR}." + prompt ans "Press ENTER when it's done or no additional dependencies. " + fi + + az login --use-device-code -o none + report_status "$?" "login" + + # Start provisioning + + if [ $(az group exists -n $RESOURCE_GROUP) == 'false' ] + then + echo "Creating Resource Group $RESOURCE_GROUP at Location $LOCATION" + az group create --output none --name $RESOURCE_GROUP --location $LOCATION + report_status "$?" "creating resource group" + elif [ $useDefault = true ] + then + report_status "1" "Only one NVFL server VM and its resource group is allowed. $RESOURCE_GROUP exists and thus creating duplicate resource group" + else + echo "Users require to reuse Resource Group $RESOURCE_GROUP. This script will modify the group and may not work always." + fi + + echo "Creating Virtual Machine, will take a few minutes" + if [ $self_dns = true ] + then + az vm create \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --location $LOCATION \ + --name $VM_NAME \ + --image $VM_IMAGE \ + --size $VM_SIZE \ + --admin-username $ADMIN_USERNAME \ + --admin-password $PASSWORD \ + --authentication-type password \ + --public-ip-address nvflare_server_ip \ + --public-ip-address-allocation static \ + --public-ip-sku Standard > /tmp/vm.json + else + az vm create \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --location $LOCATION \ + --name $VM_NAME \ + --image $VM_IMAGE \ + --size $VM_SIZE \ + --admin-username $ADMIN_USERNAME \ + --admin-password $PASSWORD \ + --authentication-type password \ + --public-ip-address nvflare_server_ip \ + --public-ip-address-allocation static \ + --public-ip-sku Standard \ + --public-ip-address-dns-name $DNS_TAG > /tmp/vm.json + fi + report_status "$?" "creating virtual machine" + + IP_ADDRESS=$(jq -r .publicIpAddress /tmp/vm.json) + echo "Setting up network related configuration" + az network nsg create \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --location $LOCATION \ + --name $NSG_NAME + report_status "$?" "creating network security group" + + az network nsg rule create \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --name SSH \ + --nsg-name $NSG_NAME \ + --priority 1000 \ + --protocol Tcp \ + --destination-port-ranges 22 + report_status "$?" "creating network security group rule for SSH" + + az network nsg rule create \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --name FL_PORT \ + --nsg-name $NSG_NAME \ + --priority 1001 \ + --protocol Tcp \ + --destination-port-ranges $FL_PORT + report_status "$?" "creating network security group rule for FL port" + + az network nsg rule create \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --name ADMIN_PORT \ + --nsg-name $NSG_NAME \ + --priority 1002 \ + --protocol Tcp \ + --destination-port-ranges $ADMIN_PORT + report_status "$?" "creating network security group rule for Admin port" + +azure_start_cln_header_sh: | + RESOURCE_GROUP=nvflare_client_rg_${RANDOM}_${RANDOM} + VM_NAME=nvflare_client + VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest + VM_SIZE=Standard_B2ms + NSG_NAME=nvflare_nsgc + ADMIN_USERNAME=nvflare + PASSWORD="NVFl@r3_P@88"$RANDOM"w0rd" + DEST_FOLDER=/var/tmp/cloud + LOCATION=westus2 + NIC_NAME=${VM_NAME}VMNic + echo "This script requires az (Azure CLI), sshpass and jq. Now checking if they are installed." + + check_binary az "Please see https://learn.microsoft.com/en-us/cli/azure/install-azure-cli on how to install it on your system." + check_binary sshpass "Please install it first." + check_binary jq "Please install it first." + + + if [ -z ${image_name+x} ] + then + container=false + else + container=true + fi + + if [ $container = true ] + then + VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest + VM_SIZE=Standard_D8s_v3 + else + VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest + VM_SIZE=Standard_B2ms + fi + if [ -z ${config_file+x} ] + then + useDefault=true + else + useDefault=false + . $config_file + report_status "$?" "Loading config file" + fi + + if [ $useDefault = true ] + then + while true + do + prompt LOCATION "Cloud location, press ENTER to accept default ${LOCATION}: " + prompt VM_IMAGE "Cloud VM image, press ENTER to accept default ${VM_IMAGE}: " + prompt VM_SIZE "Cloud VM size, press ENTER to accept default ${VM_SIZE}: " + prompt ans "location = ${LOCATION}, VM image = ${VM_IMAGE}, VM size = ${VM_SIZE}, OK? (Y/n) " + if [[ $ans = "" ]] || [[ $ans =~ ^(y|Y)$ ]]; then break; fi + done + fi + + if [ $container = false ] + then + echo "If the client requires additional dependencies, please copy the requirements.txt to ${DIR}." + prompt ans "Press ENTER when it's done or no additional dependencies. " + fi + + az login --use-device-code -o none + report_status "$?" "login" + + # Start provisioning + + if [ $(az group exists -n $RESOURCE_GROUP) == 'false' ] + then + echo "Creating Resource Group $RESOURCE_GROUP at Location $LOCATION" + az group create --output none --name $RESOURCE_GROUP --location $LOCATION + report_status "$?" "creating resource group" + else + echo "Resource Group $RESOURCE_GROUP exists, will reuse it." + fi + + echo "Creating Virtual Machine, will take a few minutes" + az vm create \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --location $LOCATION \ + --name $VM_NAME \ + --image $VM_IMAGE \ + --size $VM_SIZE \ + --admin-username $ADMIN_USERNAME \ + --admin-password $PASSWORD \ + --authentication-type password \ + --public-ip-sku Standard > /tmp/vm.json + report_status "$?" "creating virtual machine" + + IP_ADDRESS=$(jq -r .publicIpAddress /tmp/vm.json) + + echo "Setting up network related configuration" + + az network nsg create \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --location $LOCATION \ + --name $NSG_NAME + report_status "$?" "creating network security group" + + az network nsg rule create \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --name SSH \ + --nsg-name $NSG_NAME \ + --priority 1000 \ + --protocol Tcp \ + --destination-port-ranges 22 + report_status "$?" "creating network security group rule for SSH" + +azure_start_common_sh: | + az network nic update \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --name $NIC_NAME \ + --network-security-group $NSG_NAME + report_status "$?" "updating network interface card" + + echo "Copying files to $VM_NAME" + DEST=$ADMIN_USERNAME@${IP_ADDRESS}:$DEST_FOLDER + echo "Destination folder is ${DEST}" + cd $DIR/.. && sshpass -p $PASSWORD scp -r -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $PWD $DEST + report_status "$?" "copying startup kits to VM" + + if [ $container = true ] + then + echo "Installing and lauching container in $VM_NAME, may take a few minutes." + scripts=$(cat << 'EOF' + sudo apt-get update && \ + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y ca-certificates curl gnupg lsb-release && \ + sudo mkdir -p /etc/apt/keyrings && \ + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \ + echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \ + sudo apt-get update && \ + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y docker-ce docker-ce-cli containerd.io + EOF + ) + az vm run-command invoke \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --command-id RunShellScript \ + --name $VM_NAME \ + --scripts \ + "$scripts" > /tmp/docker_engine.json + report_status "$?" "installing docker engine" + az vm run-command invoke \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --command-id RunShellScript \ + --name $VM_NAME \ + --scripts \ + "sudo usermod -aG docker $ADMIN_USERNAME" >> /tmp/docker_engine.json + report_status "$?" "Setting user group" + az vm run-command invoke \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --command-id RunShellScript \ + --name $VM_NAME \ + --scripts \ + "docker run -d -v ${DEST_FOLDER}:${DEST_FOLDER} {~~docker_network~~} ${image_name} /bin/bash -c \"python -u -m nvflare.private.fed.app.{~~type~~}.{~~type~~}_train -m ${DEST_FOLDER} -s fed_{~~type~~}.json --set {~~cln_uid~~} secure_train=true config_folder=config org={~~ORG~~} \" " > /tmp/vm_create.json 2>&1 + report_status "$?" "launching container" + else + echo "Installing packages in $VM_NAME, may take a few minutes." + az vm run-command invoke \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --command-id RunShellScript \ + --name $VM_NAME \ + --scripts \ + "echo ${DEST_FOLDER} && wget -q https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && python3 -m pip install --ignore-installed nvflare && touch ${DEST_FOLDER}/startup/requirements.txt && python3 -m pip install -r ${DEST_FOLDER}/startup/requirements.txt && ${DEST_FOLDER}/startup/start.sh && sleep 20 && cat ${DEST_FOLDER}/log.txt" > /tmp/vm_create.json + report_status "$?" "installing packages" + fi + echo "System was provisioned" + echo "To delete the resource group (also delete the VM), run the following command" + echo "az group delete -n ${RESOURCE_GROUP}" + echo "To login to the VM with SSH, use ${ADMIN_USERNAME} : ${PASSWORD}" > vm_credential.txt + +azure_start_dsb_sh: | + RESOURCE_GROUP=nvflare_dashboard_rg_${RANDOM}_${RANDOM} + VM_NAME=nvflare_dashboard + VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest + VM_SIZE=Standard_B2ms + NSG_NAME=nvflare_nsgc + ADMIN_USERNAME=nvflare + PASSWORD="NVFl@r3_P@88"$RANDOM"w0rd" + DEST_FOLDER=/var/tmp/cloud + LOCATION=westus2 + NIC_NAME=${VM_NAME}VMNic + + echo "This script requires az (Azure CLI), sshpass and jq. Now checking if they are installed." + + check_binary az "Please see https://learn.microsoft.com/en-us/cli/azure/install-azure-cli on how to install it on your system." + check_binary sshpass "Please install it first." + check_binary jq "Please install it first." + + echo "One initial user will be created when starting dashboard." + echo "Please enter the email address for this user." + read email + credential="${email}:$RANDOM" + + az login --use-device-code -o none + report_status "$?" "login" + + # Start provisioning + if [ $(az group exists -n $RESOURCE_GROUP) == 'false' ] + then + echo "Creating Resource Group $RESOURCE_GROUP at Location $LOCATION" + az group create --output none --name $RESOURCE_GROUP --location $LOCATION + report_status "$?" "creating resource group" + else + echo "Resource Group $RESOURCE_GROUP exists, will reuse it." + fi + + echo "Creating Virtual Machine, will take a few minutes" + az vm create \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --location $LOCATION \ + --name $VM_NAME \ + --image $VM_IMAGE \ + --size $VM_SIZE \ + --admin-username $ADMIN_USERNAME \ + --admin-password $PASSWORD \ + --authentication-type password \ + --public-ip-sku Standard > /tmp/vm.json + report_status "$?" "creating virtual machine" + + IP_ADDRESS=$(jq -r .publicIpAddress /tmp/vm.json) + report_status "$?" "extracting ip address" + + echo "Setting up network related configuration" + az network nsg create \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --location $LOCATION \ + --name $NSG_NAME + report_status "$?" "creating network security group" + + az network nsg rule create \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --name SSH \ + --nsg-name $NSG_NAME \ + --priority 1000 \ + --protocol Tcp \ + --destination-port-ranges 22 + report_status "$?" "creating network security group rule for SSH" + + az network nsg rule create \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --name HTTPS \ + --nsg-name $NSG_NAME \ + --priority 1001 \ + --protocol Tcp \ + --destination-port-ranges 443 + report_status "$?" "creating network security group rule for HTTPS" + + az network nic update \ + --output none \ + --resource-group $RESOURCE_GROUP \ + --name $NIC_NAME \ + --network-security-group $NSG_NAME + report_status "$?" "updating network interface card" + + echo "Installing docker engine in $VM_NAME, may take a few minutes." + scripts=$(cat << 'EOF' + sudo apt-get update && \ + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y ca-certificates curl gnupg lsb-release && \ + sudo mkdir -p /etc/apt/keyrings && \ + curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \ + echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \ + sudo apt-get update && \ + sudo DEBIAN_FRONTEND=noninteractive apt-get install -y docker-ce docker-ce-cli containerd.io + EOF + ) + az vm run-command invoke \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --command-id RunShellScript \ + --name $VM_NAME \ + --scripts \ + "$scripts" > /tmp/docker_engine.json + report_status "$?" "installing docker engine" + az vm run-command invoke \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --command-id RunShellScript \ + --name $VM_NAME \ + --scripts \ + "sudo usermod -aG docker $ADMIN_USERNAME" >> /tmp/docker_engine.json + report_status "$?" "installing docker engine" + + DEST_FOLDER=/home/${ADMIN_USERNAME} + echo "Installing nvflare in $VM_NAME, may take a few minutes." + az vm run-command invoke \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --command-id RunShellScript \ + --name $VM_NAME \ + --scripts \ + "echo ${DEST_FOLDER} && wget -q https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && python3 -m pip install --ignore-installed {~~NVFLARE~~} && mkdir -p ${DEST_FOLDER}/cert && chown -R ${ADMIN_USERNAME} ${DEST_FOLDER}" > /tmp/nvflare.json + report_status "$?" "installing nvflare" + + echo "Checking if certificate (web.crt) and private key (web.key) are available" + if [[ -f "web.crt" && -f "web.key" ]]; then + DEST=$ADMIN_USERNAME@$IP_ADDRESS:${DEST_FOLDER}/cert + echo "Destination folder is ${DEST}" + sshpass -p $PASSWORD scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null web.{crt,key} $DEST + report_status "$?" "copying cert/key to VM ${DEST} folder" + secure=true + else + echo "No web.crt and web.key found" + secure=false + fi + + echo "Starting dashboard" + az vm run-command invoke \ + --output json \ + --resource-group $RESOURCE_GROUP \ + --command-id RunShellScript \ + --name $VM_NAME \ + --scripts \ + "cd ${DEST_FOLDER} && python3 -m nvflare.dashboard.cli --start -f ${DEST_FOLDER} --cred ${credential} {~~START_OPT~~}" > /tmp/dashboard.json + + # credential=$(jq -r .value[0].message /tmp/dashboard.json | grep "Project admin") + # echo "The VM was created with user: ${ADMIN_USERNAME} and password: ${PASSWORD}" + if [ "$secure" = true ] + then + echo "URL is https://${IP_ADDRESS}" + else + echo "URL is http://${IP_ADDRESS}:443" + fi + echo "Note: you may need to configure DNS server with your DNS hostname and the above IP address." + echo "Project admin credential (username:password) is ${credential} ." + echo "To stop the dashboard, run az group delete -n ${RESOURCE_GROUP}" + echo "To login to the VM with SSH, use ${ADMIN_USERNAME} : ${PASSWORD}" > vm_credential.txt diff --git a/nvflare/lighter/impl/master_template.yml b/nvflare/lighter/impl/master_template.yml index 5d816858c6..24342030ba 100644 --- a/nvflare/lighter/impl/master_template.yml +++ b/nvflare/lighter/impl/master_template.yml @@ -417,7 +417,7 @@ stop_fl_sh: | ;; esac -sub_start_cln_sh: | +sub_start_sh: | #!/usr/bin/env bash DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" echo "WORKSPACE set to $DIR/.." @@ -440,7 +440,7 @@ sub_start_cln_sh: | exit fi lst=$SECONDS - ((python3 -u -m nvflare.private.fed.app.client.client_train -m $DIR/.. -s fed_client.json --set secure_train=true uid={~~client_name~~} org={~~org_name~~} config_folder={~~config_folder~~} 2>&1 & echo $! >&3 ) 3>$DIR/../pid.fl ) + ((python3 -u -m nvflare.private.fed.app.{~~type~~}.{~~type~~}_train -m $DIR/.. -s fed_{~~type~~}.json --set secure_train=true {~~cln_uid~~} org={~~org_name~~} config_folder={~~config_folder~~} 2>&1 & echo $! >&3 ) 3>$DIR/../pid.fl ) pid=`cat $DIR/../pid.fl` echo "new pid ${pid}" } @@ -506,93 +506,6 @@ sub_start_cln_sh: | rm -f $DIR/../pid.fl $DIR/../shutdown.fl $DIR/../restart.fl $DIR/../daemon_pid.fl -sub_start_svr_sh: | - #!/usr/bin/env bash - DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" - echo "WORKSPACE set to $DIR/.." - mkdir -p $DIR/../transfer - - SECONDS=0 - lst=-400 - restart_count=0 - start_fl() { - if [[ $(( $SECONDS - $lst )) -lt 300 ]]; then - ((restart_count++)) - else - restart_count=0 - fi - if [[ $(($SECONDS - $lst )) -lt 300 && $restart_count -ge 5 ]]; then - echo "System is in trouble and unable to start the task!!!!!" - rm -f $DIR/../pid.fl $DIR/../shutdown.fl $DIR/../restart.fl $DIR/../daemon_pid.fl - exit - fi - lst=$SECONDS - ((python3 -u -m nvflare.private.fed.app.server.server_train -m $DIR/.. -s fed_server.json --set secure_train=true org={~~org_name~~} config_folder={~~config_folder~~} 2>&1 & echo $! >&3 ) 3>$DIR/../pid.fl ) - pid=`cat $DIR/../pid.fl` - echo "new pid ${pid}" - } - - stop_fl() { - if [[ ! -f "$DIR/../pid.fl" ]]; then - echo "No pid.fl. No need to kill process." - return - fi - pid=`cat $DIR/../pid.fl` - sleep 5 - kill -0 ${pid} 2> /dev/null 1>&2 - if [[ $? -ne 0 ]]; then - echo "Process already terminated" - return - fi - kill -9 $pid - rm -f $DIR/../pid.fl $DIR/../shutdown.fl $DIR/../restart.fl - } - - if [[ -f "$DIR/../daemon_pid.fl" ]]; then - dpid=`cat $DIR/../daemon_pid.fl` - kill -0 ${dpid} 2> /dev/null 1>&2 - if [[ $? -eq 0 ]]; then - echo "There seems to be one instance, pid=$dpid, running." - echo "If you are sure it's not the case, please kill process $dpid and then remove daemon_pid.fl in $DIR/.." - exit - fi - rm -f $DIR/../daemon_pid.fl - fi - - echo $BASHPID > $DIR/../daemon_pid.fl - - while true - do - sleep 5 - if [[ ! -f "$DIR/../pid.fl" ]]; then - echo "start fl because of no pid.fl" - start_fl - continue - fi - pid=`cat $DIR/../pid.fl` - kill -0 ${pid} 2> /dev/null 1>&2 - if [[ $? -ne 0 ]]; then - if [[ -f "$DIR/../shutdown.fl" ]]; then - echo "Gracefully shutdown." - break - fi - echo "start fl because process of ${pid} does not exist" - start_fl - continue - fi - if [[ -f "$DIR/../shutdown.fl" ]]; then - echo "About to shutdown." - stop_fl - break - fi - if [[ -f "$DIR/../restart.fl" ]]; then - echo "About to restart." - stop_fl - fi - done - - rm -f $DIR/../pid.fl $DIR/../shutdown.fl $DIR/../restart.fl $DIR/../daemon_pid.fl - docker_cln_sh: | #!/usr/bin/env bash DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )" @@ -910,589 +823,6 @@ cloud_script_header: | shift done -azure_start_svr_sh: | - RESOURCE_GROUP=nvflare_rg - VM_NAME=nvflare_server - VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest - VM_SIZE=Standard_B2ms - NSG_NAME=nvflare_nsgs - ADMIN_USERNAME=nvflare - PASSWORD="NVFl@r3_P@88"$RANDOM"w0rd" - DEST_FOLDER=/var/tmp/cloud - NIC_NAME=${VM_NAME}VMNic - SERVER_NAME={~~server_name~~} - FL_PORT=8002 - ADMIN_PORT=8003 - - echo "This script requires az (Azure CLI), sshpass and jq. Now checking if they are installed." - - check_binary az "Please see https://learn.microsoft.com/en-us/cli/azure/install-azure-cli on how to install it on your system." - check_binary sshpass "Please install it first." - check_binary jq "Please install it first." - - self_dns=true - if [[ "$SERVER_NAME" = *".cloudapp.azure.com"* ]] - then - DNS_TAG=$(echo $SERVER_NAME | cut -d "." -f 1) - DERIVED_LOCATION=$(echo $SERVER_NAME | cut -d "." -f 2) - LOCATION=$DERIVED_LOCATION - self_dns=false - else - echo "Warning: ${SERVER_NAME} does not end with .cloudapp.azure.com." - echo "The cloud launch process will not create the domain name for you." - echo "Please use your own DNS to set the information." - LOCATION=westus2 - fi - - if [ -z ${image_name+x} ] - then - container=false - else - container=true - fi - - if [ $container = true ] - then - VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest - VM_SIZE=Standard_D8s_v3 - else - VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest - VM_SIZE=Standard_B2ms - fi - - if [ -z ${config_file+x} ] - then - useDefault=true - else - useDefault=false - . $config_file - report_status "$?" "Loading config file" - if [ $self_dns = false ] && [ $DERIVED_LOCATION != $LOCATION ] - then - echo "Server name implies LOCATION=${DERIVED_LOCATION} but the config file specifies LOCATION=${LOCATION}. Unable to continue provisioning." - exit 1 - fi - fi - - if [ $useDefault = true ] - then - while true - do - prompt VM_IMAGE "Cloud VM image, press ENTER to accept default ${VM_IMAGE}: " - prompt VM_SIZE "Cloud VM size, press ENTER to accept default ${VM_SIZE}: " - if [ $self_dns = true ] - then - prompt LOCATION "Cloud location, press ENTER to accept default ${LOCATION}: " - prompt ans "VM image = ${VM_IMAGE}, VM size = ${VM_SIZE}, location = ${LOCATION}, OK? (Y/n) " - else - prompt ans "VM image = ${VM_IMAGE}, VM size = ${VM_SIZE}, OK? (Y/n) " - fi - if [[ $ans = "" ]] || [[ $ans =~ ^(y|Y)$ ]]; then break; fi - done - fi - - if [ $container = false ] - then - echo "If the client requires additional dependencies, please copy the requirements.txt to ${DIR}." - prompt ans "Press ENTER when it's done or no additional dependencies. " - fi - - az login --use-device-code -o none - report_status "$?" "login" - - # Start provisioning - - if [ $(az group exists -n $RESOURCE_GROUP) == 'false' ] - then - echo "Creating Resource Group $RESOURCE_GROUP at Location $LOCATION" - az group create --output none --name $RESOURCE_GROUP --location $LOCATION - report_status "$?" "creating resource group" - elif [ $useDefault = true ] - then - report_status "1" "Only one NVFL server VM and its resource group is allowed. $RESOURCE_GROUP exists and thus creating duplicate resource group" - else - echo "Users require to reuse Resource Group $RESOURCE_GROUP. This script will modify the group and may not work always." - fi - - echo "Creating Virtual Machine, will take a few minutes" - if [ $self_dns = true ] - then - az vm create \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --location $LOCATION \ - --name $VM_NAME \ - --image $VM_IMAGE \ - --size $VM_SIZE \ - --admin-username $ADMIN_USERNAME \ - --admin-password $PASSWORD \ - --authentication-type password \ - --public-ip-address nvflare_server_ip \ - --public-ip-address-allocation static \ - --public-ip-sku Standard > /tmp/vm.json - else - az vm create \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --location $LOCATION \ - --name $VM_NAME \ - --image $VM_IMAGE \ - --size $VM_SIZE \ - --admin-username $ADMIN_USERNAME \ - --admin-password $PASSWORD \ - --authentication-type password \ - --public-ip-address nvflare_server_ip \ - --public-ip-address-allocation static \ - --public-ip-sku Standard \ - --public-ip-address-dns-name $DNS_TAG > /tmp/vm.json - fi - report_status "$?" "creating virtual machine" - - IP_ADDRESS=$(jq -r .publicIpAddress /tmp/vm.json) - echo "Setting up network related configuration" - az network nsg create \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --location $LOCATION \ - --name $NSG_NAME - report_status "$?" "creating network security group" - - az network nsg rule create \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --name SSH \ - --nsg-name $NSG_NAME \ - --priority 1000 \ - --protocol Tcp \ - --destination-port-ranges 22 - report_status "$?" "creating network security group rule for SSH" - - az network nsg rule create \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --name FL_PORT \ - --nsg-name $NSG_NAME \ - --priority 1001 \ - --protocol Tcp \ - --destination-port-ranges $FL_PORT - report_status "$?" "creating network security group rule for FL port" - - az network nsg rule create \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --name ADMIN_PORT \ - --nsg-name $NSG_NAME \ - --priority 1002 \ - --protocol Tcp \ - --destination-port-ranges $ADMIN_PORT - report_status "$?" "creating network security group rule for Admin port" - - az network nic update \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --name $NIC_NAME \ - --network-security-group $NSG_NAME - report_status "$?" "updating network interface card" - - echo "Copying files to $VM_NAME" - DEST=$ADMIN_USERNAME@${IP_ADDRESS}:$DEST_FOLDER - echo "Destination folder is ${DEST}" - cd $DIR/.. && sshpass -p $PASSWORD scp -r -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $PWD $DEST - report_status "$?" "copying startup kits to VM" - - if [ $container = true ] - then - echo "Installing and lauching container in $VM_NAME, may take a few minutes." - scripts=$(cat << 'EOF' - sudo apt-get update && \ - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y ca-certificates curl gnupg lsb-release && \ - sudo mkdir -p /etc/apt/keyrings && \ - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \ - echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ - $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \ - sudo apt-get update && \ - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y docker-ce docker-ce-cli containerd.io - EOF - ) - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "$scripts" > /tmp/docker_engine.json - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "sudo usermod -aG docker $ADMIN_USERNAME" >> /tmp/docker_engine.json - report_status "$?" "installing docker engine" - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "docker run -d -v ${DEST_FOLDER}:${DEST_FOLDER} --network host ${DOCKER_OPTION} ${image_name} /bin/bash -c \"python -u -m nvflare.private.fed.app.server.server_train -m ${DEST_FOLDER} -s fed_server.json --set secure_train=true config_folder=config org={~~ORG~~} \" " > /tmp/vm_create.json 2>&1 - report_status "$?" "launching container" - else - echo "Installing packages in $VM_NAME, may take a few minutes." - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "echo ${DEST_FOLDER} && wget -q https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && python3 -m pip install --ignore-installed nvflare && touch ${DEST_FOLDER}/startup/requirements.txt && python3 -m pip install -r ${DEST_FOLDER}/startup/requirements.txt && ${DEST_FOLDER}/startup/start.sh && sleep 20 && cat ${DEST_FOLDER}/log.txt" > /tmp/vm_create.json - report_status "$?" "installing packages" - fi - echo "System was provisioned" - echo "To delete the resource group (also delete the VM), run the following command" - echo "az group delete -n ${RESOURCE_GROUP}" - echo "To login to the VM with SSH, use ${ADMIN_USERNAME} : ${PASSWORD}" > vm_credential.txt - -azure_start_cln_sh: | - RESOURCE_GROUP=nvflare_client_rg_${RANDOM}_${RANDOM} - VM_NAME=nvflare_client - VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest - VM_SIZE=Standard_B2ms - NSG_NAME=nvflare_nsgc - ADMIN_USERNAME=nvflare - PASSWORD="NVFl@r3_P@88"$RANDOM"w0rd" - DEST_FOLDER=/var/tmp/cloud - LOCATION=westus2 - NIC_NAME=${VM_NAME}VMNic - echo "This script requires az (Azure CLI), sshpass and jq. Now checking if they are installed." - - check_binary az "Please see https://learn.microsoft.com/en-us/cli/azure/install-azure-cli on how to install it on your system." - check_binary sshpass "Please install it first." - check_binary jq "Please install it first." - - - if [ -z ${image_name+x} ] - then - container=false - else - container=true - fi - - if [ $container = true ] - then - VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest - VM_SIZE=Standard_D8s_v3 - else - VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest - VM_SIZE=Standard_B2ms - fi - if [ -z ${config_file+x} ] - then - useDefault=true - else - useDefault=false - . $config_file - report_status "$?" "Loading config file" - fi - - if [ $useDefault = true ] - then - while true - do - prompt LOCATION "Cloud location, press ENTER to accept default ${LOCATION}: " - prompt VM_IMAGE "Cloud VM image, press ENTER to accept default ${VM_IMAGE}: " - prompt VM_SIZE "Cloud VM size, press ENTER to accept default ${VM_SIZE}: " - prompt ans "location = ${LOCATION}, VM image = ${VM_IMAGE}, VM size = ${VM_SIZE}, OK? (Y/n) " - if [[ $ans = "" ]] || [[ $ans =~ ^(y|Y)$ ]]; then break; fi - done - fi - - if [ $container = false ] - then - echo "If the client requires additional dependencies, please copy the requirements.txt to ${DIR}." - prompt ans "Press ENTER when it's done or no additional dependencies. " - fi - - az login --use-device-code -o none - report_status "$?" "login" - - # Start provisioning - - if [ $(az group exists -n $RESOURCE_GROUP) == 'false' ] - then - echo "Creating Resource Group $RESOURCE_GROUP at Location $LOCATION" - az group create --output none --name $RESOURCE_GROUP --location $LOCATION - report_status "$?" "creating resource group" - else - echo "Resource Group $RESOURCE_GROUP exists, will reuse it." - fi - - echo "Creating Virtual Machine, will take a few minutes" - az vm create \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --location $LOCATION \ - --name $VM_NAME \ - --image $VM_IMAGE \ - --size $VM_SIZE \ - --admin-username $ADMIN_USERNAME \ - --admin-password $PASSWORD \ - --authentication-type password \ - --public-ip-sku Standard > /tmp/vm.json - report_status "$?" "creating virtual machine" - - IP_ADDRESS=$(jq -r .publicIpAddress /tmp/vm.json) - - echo "Setting up network related configuration" - - az network nsg create \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --location $LOCATION \ - --name $NSG_NAME - report_status "$?" "creating network security group" - - az network nsg rule create \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --name SSH \ - --nsg-name $NSG_NAME \ - --priority 1000 \ - --protocol Tcp \ - --destination-port-ranges 22 - report_status "$?" "creating network security group rule for SSH" - - az network nic update \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --name $NIC_NAME \ - --network-security-group $NSG_NAME - report_status "$?" "updating network interface card" - - echo "Copying files to $VM_NAME" - DEST=$ADMIN_USERNAME@$IP_ADDRESS:$DEST_FOLDER - echo "Destination folder is ${DEST}" - cd $DIR/.. && sshpass -p $PASSWORD scp -r -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $PWD $DEST - report_status "$?" "copying startup kits to VM" - - if [ $container = true ] - then - echo "Installing and lauching container in $VM_NAME, may take a few minutes." - scripts=$(cat <<- 'EOF' - sudo apt-get update && \ - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y ca-certificates curl gnupg lsb-release && \ - sudo mkdir -p /etc/apt/keyrings && \ - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \ - echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ - $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \ - sudo apt-get update && \ - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y docker-ce docker-ce-cli containerd.io - EOF - ) - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "$scripts" > /tmp/docker_engine.json - report_status "$?" "installing docker engine" - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "sudo usermod -aG docker $ADMIN_USERNAME" >> /tmp/docker_engine.json - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "docker run -d -v ${DEST_FOLDER}:${DEST_FOLDER} ${image_name} /bin/bash -c \"python -u -m nvflare.private.fed.app.client.client_train -m ${DEST_FOLDER} -s fed_client.json --set uid={~~SITE~~} secure_train=true config_folder=config org={~~ORG~~} \" " > /tmp/vm_create.json 2>&1 - report_status "$?" "launching container" - else - echo "Installing packages in $VM_NAME, may take a few minutes." - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "echo ${DEST_FOLDER} && wget -q https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && python3 -m pip install --ignore-installed nvflare && touch ${DEST_FOLDER}/startup/requirements.txt && python3 -m pip install -r ${DEST_FOLDER}/startup/requirements.txt && ${DEST_FOLDER}/startup/start.sh && sleep 20 && cat ${DEST_FOLDER}/log.txt" > /tmp/vm_create.json - report_status "$?" "installing packages" - fi - echo "System was provisioned" - echo "To delete the resource group (also delete the VM), run the following command" - echo "az group delete -n ${RESOURCE_GROUP}" - echo "To login to the VM with SSH, use ${ADMIN_USERNAME} : ${PASSWORD}" > vm_credential.txt - -azure_start_dsb_sh: | - RESOURCE_GROUP=nvflare_dashboard_rg_${RANDOM}_${RANDOM} - VM_NAME=nvflare_dashboard - VM_IMAGE=Canonical:0001-com-ubuntu-server-focal:20_04-lts-gen2:latest - VM_SIZE=Standard_B2ms - NSG_NAME=nvflare_nsgc - ADMIN_USERNAME=nvflare - PASSWORD="NVFl@r3_P@88"$RANDOM"w0rd" - DEST_FOLDER=/var/tmp/cloud - LOCATION=westus2 - NIC_NAME=${VM_NAME}VMNic - - echo "This script requires az (Azure CLI), sshpass and jq. Now checking if they are installed." - - check_binary az "Please see https://learn.microsoft.com/en-us/cli/azure/install-azure-cli on how to install it on your system." - check_binary sshpass "Please install it first." - check_binary jq "Please install it first." - - echo "One initial user will be created when starting dashboard." - echo "Please enter the email address for this user." - read email - credential="${email}:$RANDOM" - - az login --use-device-code -o none - report_status "$?" "login" - - # Start provisioning - if [ $(az group exists -n $RESOURCE_GROUP) == 'false' ] - then - echo "Creating Resource Group $RESOURCE_GROUP at Location $LOCATION" - az group create --output none --name $RESOURCE_GROUP --location $LOCATION - report_status "$?" "creating resource group" - else - echo "Resource Group $RESOURCE_GROUP exists, will reuse it." - fi - - echo "Creating Virtual Machine, will take a few minutes" - az vm create \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --location $LOCATION \ - --name $VM_NAME \ - --image $VM_IMAGE \ - --size $VM_SIZE \ - --admin-username $ADMIN_USERNAME \ - --admin-password $PASSWORD \ - --authentication-type password \ - --public-ip-sku Standard > /tmp/vm.json - report_status "$?" "creating virtual machine" - - IP_ADDRESS=$(jq -r .publicIpAddress /tmp/vm.json) - report_status "$?" "extracting ip address" - - echo "Setting up network related configuration" - az network nsg create \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --location $LOCATION \ - --name $NSG_NAME - report_status "$?" "creating network security group" - - az network nsg rule create \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --name SSH \ - --nsg-name $NSG_NAME \ - --priority 1000 \ - --protocol Tcp \ - --destination-port-ranges 22 - report_status "$?" "creating network security group rule for SSH" - - az network nsg rule create \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --name HTTPS \ - --nsg-name $NSG_NAME \ - --priority 1001 \ - --protocol Tcp \ - --destination-port-ranges 443 - report_status "$?" "creating network security group rule for HTTPS" - - az network nic update \ - --output none \ - --resource-group $RESOURCE_GROUP \ - --name $NIC_NAME \ - --network-security-group $NSG_NAME - report_status "$?" "updating network interface card" - - echo "Installing docker engine in $VM_NAME, may take a few minutes." - scripts=$(cat << 'EOF' - sudo apt-get update && \ - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y ca-certificates curl gnupg lsb-release && \ - sudo mkdir -p /etc/apt/keyrings && \ - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \ - echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ - $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \ - sudo apt-get update && \ - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y docker-ce docker-ce-cli containerd.io - EOF - ) - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "$scripts" > /tmp/docker_engine.json - report_status "$?" "installing docker engine" - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "sudo usermod -aG docker $ADMIN_USERNAME" >> /tmp/docker_engine.json - report_status "$?" "installing docker engine" - - DEST_FOLDER=/home/${ADMIN_USERNAME} - echo "Installing nvflare in $VM_NAME, may take a few minutes." - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "echo ${DEST_FOLDER} && wget -q https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && python3 -m pip install --ignore-installed {~~NVFLARE~~} && mkdir -p ${DEST_FOLDER}/cert && chown -R ${ADMIN_USERNAME} ${DEST_FOLDER}" > /tmp/nvflare.json - report_status "$?" "installing nvflare" - - echo "Checking if certificate (web.crt) and private key (web.key) are available" - if [[ -f "web.crt" && -f "web.key" ]]; then - DEST=$ADMIN_USERNAME@$IP_ADDRESS:${DEST_FOLDER}/cert - echo "Destination folder is ${DEST}" - sshpass -p $PASSWORD scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null web.{crt,key} $DEST - report_status "$?" "copying cert/key to VM ${DEST} folder" - secure=true - else - echo "No web.crt and web.key found" - secure=false - fi - - echo "Starting dashboard" - az vm run-command invoke \ - --output json \ - --resource-group $RESOURCE_GROUP \ - --command-id RunShellScript \ - --name $VM_NAME \ - --scripts \ - "cd ${DEST_FOLDER} && python3 -m nvflare.dashboard.cli --start -f ${DEST_FOLDER} --cred ${credential} {~~START_OPT~~}" > /tmp/dashboard.json - - # credential=$(jq -r .value[0].message /tmp/dashboard.json | grep "Project admin") - # echo "The VM was created with user: ${ADMIN_USERNAME} and password: ${PASSWORD}" - if [ "$secure" = true ] - then - echo "URL is https://${IP_ADDRESS}" - else - echo "URL is http://${IP_ADDRESS}:443" - fi - echo "Note: you may need to configure DNS server with your DNS hostname and the above IP address." - echo "Project admin credential (username:password) is ${credential} ." - echo "To stop the dashboard, run az group delete -n ${RESOURCE_GROUP}" - echo "To login to the VM with SSH, use ${ADMIN_USERNAME} : ${PASSWORD}" > vm_credential.txt - adm_notebook: | { "cells": [ @@ -1611,402 +941,3 @@ adm_notebook: | "nbformat_minor": 5 } -aws_start_svr_sh: | - VM_NAME=nvflare_server - SECURITY_GROUP=nvflare_server_sg - DEST_FOLDER=/var/tmp/cloud - KEY_PAIR=NVFlareServerKeyPair - KEY_FILE=${KEY_PAIR}.pem - - echo "This script requires aws (AWS CLI), sshpass, dig and jq. Now checking if they are installed." - - check_binary aws "Please see https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html on how to install it on your system." - check_binary sshpass "Please install it first." - check_binary dig "Please install it first." - check_binary jq "Please install it first." - - if [ -z ${image_name+x} ] - then - container=false - else - container=true - fi - - if [ $container = true ] - then - AMI_IMAGE=ami-06b8d5099f3a8d79d - EC2_TYPE=t2.xlarge - REGION=us-west-2 - else - AMI_IMAGE=ami-04bad3c587fe60d89 - EC2_TYPE=t2.small - REGION=us-west-2 - fi - - if [ -z ${config_file+x} ] - then - useDefault=true - else - useDefault=false - . $config_file - report_status "$?" "Loading config file" - fi - - - if [ $useDefault = true ] - then - while true - do - prompt AMI_IMAGE "Cloud AMI image, press ENTER to accept default ${AMI_IMAGE}: " - prompt EC2_TYPE "Cloud EC2 type, press ENTER to accept default ${EC2_TYPE}: " - prompt REGIION "Cloud EC2 region, press ENTER to accept default ${REGION}: " - prompt ans "region = ${REGION}, ami image = ${AMI_IMAGE}, EC2 type = ${EC2_TYPE}, OK? (Y/n) " - if [[ $ans = "" ]] || [[ $ans =~ ^(y|Y)$ ]] - then - break - fi - done - fi - - if [ $container = false ] - then - echo "If the server requires additional dependencies, please copy the requirements.txt to ${DIR}." - prompt ans "Press ENTER when it's done or no additional dependencies. " - fi - - cd $DIR/.. - # Generate key pair - - echo "Generating key pair for VM" - - aws ec2 delete-key-pair --key-name $KEY_PAIR > /dev/null 2>&1 - rm -rf $KEY_FILE - aws ec2 create-key-pair --key-name $KEY_PAIR --query 'KeyMaterial' --output text > $KEY_FILE - report_status "$?" "creating key pair" - chmod 400 $KEY_FILE - - # Generate Security Group - - sg_result=$(aws ec2 create-security-group --group-name $SECURITY_GROUP --description "NVFlare security group") - report_status "$?" "Only one NVFL server VM and its security group is allowed. $SECURITY_GROUP exists and thus creating duplicate security group" - sg_id=$(echo $sg_result | jq -r .GroupId) - my_public_ip=$(dig +short myip.opendns.com @resolver1.opendns.com) - if [ "$?" -eq 0 ] && [[ "$my_public_ip" =~ ^(([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))\.){3}([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))$ ]] - then - aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr ${my_public_ip}/32 > /tmp/sec_grp.log - else - echo "getting my public IP failed, please manually configure the inbound rule to limit SSH access" - aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr 0.0.0.0/0 > /tmp/sec_grp.log - fi - aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 8002-8003 --cidr 0.0.0.0/0 >> /tmp/sec_grp.log - report_status "$?" "creating security group rules" - - # Start provisioning - - echo "Creating VM at region $REGION, may take a few minutes." - - aws ec2 run-instances --region $REGION --image-id $AMI_IMAGE --count 1 --instance-type $EC2_TYPE --key-name $KEY_PAIR --security-group-ids $sg_id > vm_create.json - report_status "$?" "creating VM" - instance_id=$(jq -r .Instances[0].InstanceId vm_create.json) - - aws ec2 wait instance-status-ok --instance-ids $instance_id - aws ec2 describe-instances --instance-ids $instance_id > vm_result.json - - IP_ADDRESS=$(jq -r .Reservations[0].Instances[0].PublicIpAddress vm_result.json) - - echo "VM created with IP address: ${IP_ADDRESS}" - - echo "Copying files to $VM_NAME" - DEST_SITE=ubuntu@${IP_ADDRESS} - DEST=${DEST_SITE}:${DEST_FOLDER} - echo "Destination folder is ${DEST}" - scp -q -i $KEY_FILE -r -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $PWD $DEST - report_status "$?" "copying startup kits to VM" - - if [ $container = true ] - then - echo "Launching container with docker option ${DOCKER_OPTION}." - ssh -f -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ - "docker run -d -v ${DEST_FOLDER}:${DEST_FOLDER} --network host ${DOCKER_OPTION} ${image_name} \ - /bin/bash -c \"python -u -m nvflare.private.fed.app.server.server_train -m ${DEST_FOLDER} \ - -s fed_server.json --set secure_train=true config_folder=config org={~~ORG~~} \" " > /tmp/nvflare.log 2>&1 - report_status "$?" "launching container" - else - ssh -f -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ - "pwd && wget -q https://bootstrap.pypa.io/get-pip.py && \ - python3 get-pip.py && python3 -m pip install nvflare && \ - touch ${DEST_FOLDER}/startup/requirements.txt && \ - python3 -m pip install -r ${DEST_FOLDER}/startup/requirements.txt && \ - nohup ${DEST_FOLDER}/startup/start.sh && sleep 20 && \ - exit" > /tmp/nvflare.log 2>&1 - report_status "$?" "installing packages" - fi - - echo "System was provisioned" - echo "To terminate the EC2 instance, run the following command." - echo "aws ec2 terminate-instances --instance-ids ${instance_id}" - echo "Other resources provisioned" - echo "security group: ${SECURITY_GROUP}" - echo "key pair: ${KEY_PAIR}" - -aws_start_cln_sh: | - VM_NAME=nvflare_client - SECURITY_GROUP=nvflare_client_sg_$RANDOM - DEST_FOLDER=/var/tmp/cloud - KEY_PAIR=NVFlareClientKeyPair - KEY_FILE=${KEY_PAIR}.pem - - echo "This script requires aws (AWS CLI), sshpass, dig and jq. Now checking if they are installed." - - check_binary aws "Please see https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html on how to install it on your system." - check_binary sshpass "Please install it first." - check_binary dig "Please install it first." - check_binary jq "Please install it first." - - if [ -z ${image_name+x} ] - then - container=false - else - container=true - fi - - if [ $container = true ] - then - AMI_IMAGE=ami-06b8d5099f3a8d79d - EC2_TYPE=t2.xlarge - REGION=us-west-2 - else - AMI_IMAGE=ami-04bad3c587fe60d89 - EC2_TYPE=t2.small - REGION=us-west-2 - fi - - if [ -z ${config_file+x} ] - then - useDefault=true - else - useDefault=false - . $config_file - report_status "$?" "Loading config file" - fi - - if [ $useDefault = true ] - then - while true - do - prompt AMI_IMAGE "Cloud AMI image, press ENTER to accept default ${AMI_IMAGE}: " - prompt EC2_TYPE "Cloud EC2 type, press ENTER to accept default ${EC2_TYPE}: " - prompt REGIION "Cloud EC2 region, press ENTER to accept default ${REGION}: " - prompt ans "region = ${REGION}, ami image = ${AMI_IMAGE}, EC2 type = ${EC2_TYPE}, OK? (Y/n) " - if [[ $ans = "" ]] || [[ $ans =~ ^(y|Y)$ ]] - then - break - fi - done - fi - - if [ $container = false ] - then - echo "If the client requires additional dependencies, please copy the requirements.txt to ${DIR}." - prompt ans "Press ENTER when it's done or no additional dependencies. " - fi - - cd $DIR/.. - # Generate key pair - - echo "Generating key pair for VM" - - aws ec2 delete-key-pair --key-name $KEY_PAIR > /dev/null 2>&1 - rm -rf $KEY_FILE - aws ec2 create-key-pair --key-name $KEY_PAIR --query 'KeyMaterial' --output text > $KEY_FILE - report_status "$?" "creating key pair" - chmod 400 $KEY_FILE - - # Generate Security Group - # Try not reusing existing security group because we have to modify it for our own need. - sg_id=$(aws ec2 create-security-group --group-name $SECURITY_GROUP --description "NVFlare security group" | jq -r .GroupId) - report_status "$?" "creating security group" - my_public_ip=$(dig +short myip.opendns.com @resolver1.opendns.com) - if [ "$?" -eq 0 ] && [[ "$my_public_ip" =~ ^(([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))\.){3}([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))$ ]] - then - aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr ${my_public_ip}/32 > /tmp/sec_grp.log - else - echo "getting my public IP failed, please manually configure the inbound rule to limit SSH access" - aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr 0.0.0.0/0 > /tmp/sec_grp.log - fi - report_status "$?" "creating security group rules" - - # Start provisioning - - echo "Creating VM at region $REGION, may take a few minutes." - - aws ec2 run-instances --region $REGION --image-id $AMI_IMAGE --count 1 --instance-type $EC2_TYPE --key-name $KEY_PAIR --security-group-ids $sg_id > vm_create.json - report_status "$?" "creating VM" - instance_id=$(jq -r .Instances[0].InstanceId vm_create.json) - - aws ec2 wait instance-status-ok --instance-ids $instance_id - aws ec2 describe-instances --instance-ids $instance_id > vm_result.json - - IP_ADDRESS=$(jq -r .Reservations[0].Instances[0].PublicIpAddress vm_result.json) - - echo "VM created with IP address: ${IP_ADDRESS}" - - echo "Copying files to $VM_NAME" - DEST_SITE=ubuntu@${IP_ADDRESS} - DEST=${DEST_SITE}:${DEST_FOLDER} - echo "Destination folder is ${DEST}" - scp -q -i $KEY_FILE -r -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null $PWD $DEST - report_status "$?" "copying startup kits to VM" - - if [ $container = true ] - then - echo "Launching container with docker option ${DOCKER_OPTION}." - ssh -f -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ - "docker run -d -v ${DEST_FOLDER}:${DEST_FOLDER} --network host ${DOCKER_OPTION} ${image_name} \ - /bin/bash -c \"python -u -m nvflare.private.fed.app.client.client_train -m ${DEST_FOLDER} \ - -s fed_client.json --set uid={~~SITE~~} secure_train=true config_folder=config org={~~ORG~~} \" " > /tmp/nvflare.log 2>&1 - report_status "$?" "launching container" - else - echo "Installing packages in $VM_NAME, may take a few minutes." - ssh -f -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ - "pwd && wget -q https://bootstrap.pypa.io/get-pip.py && \ - python3 get-pip.py && python3 -m pip install nvflare && \ - touch ${DEST_FOLDER}/startup/requirements.txt && \ - python3 -m pip install -r ${DEST_FOLDER}/startup/requirements.txt && \ - nohup ${DEST_FOLDER}/startup/start.sh && sleep 20 && \ - exit" > /tmp/nvflare.log 2>&1 - - report_status "$?" "installing packages" - fi - - echo "System was provisioned" - echo "To terminate the EC2 instance, run the following command." - echo "aws ec2 terminate-instances --instance-ids ${instance_id}" - echo "Other resources provisioned" - echo "security group: ${SECURITY_GROUP}" - echo "key pair: ${KEY_PAIR}" - - -aws_start_dsb_sh: | - VM_NAME=nvflare_dashboard - AMI_IMAGE=ami-04bad3c587fe60d89 - EC2_TYPE=t2.small - SECURITY_GROUP=nvflare_dashboard_sg_$RANDOM - REGION=us-west-2 - ADMIN_USERNAME=ubuntu - DEST_FOLDER=/home/${ADMIN_USERNAME} - KEY_PAIR=NVFlareDashboardKeyPair - KEY_FILE=${KEY_PAIR}.pem - - echo "This script requires aws (AWS CLI), sshpass, dig and jq. Now checking if they are installed." - - check_binary aws "Please see https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html on how to install it on your system." - check_binary sshpass "Please install it first." - check_binary dig "Please install it first." - check_binary jq "Please install it first." - - echo "One initial user will be created when starting dashboard." - echo "Please enter the email address for this user." - read email - credential="${email}:$RANDOM" - - # Generate key pair - - echo "Generating key pair for VM" - - aws ec2 delete-key-pair --key-name $KEY_PAIR > /dev/null 2>&1 - rm -rf $KEY_FILE - aws ec2 create-key-pair --key-name $KEY_PAIR --query 'KeyMaterial' --output text > $KEY_FILE - report_status "$?" "creating key pair" - chmod 400 $KEY_FILE - - # Generate Security Group - - sg_id=$(aws ec2 create-security-group --group-name $SECURITY_GROUP --description "NVFlare security group" | jq -r .GroupId) - report_status "$?" "creating security group" - echo "Security group id: ${sg_id}" - my_public_ip=$(dig +short myip.opendns.com @resolver1.opendns.com) - if [ "$?" -eq 0 ] && [[ "$my_public_ip" =~ ^(([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))\.){3}([1-9]?[0-9]|1[0-9][0-9]|2([0-4][0-9]|5[0-5]))$ ]] - then - aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr ${my_public_ip}/32 > /tmp/sec_grp.log - else - echo "getting my public IP failed, please manually configure the inbound rule to limit SSH access" - aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 22 --cidr 0.0.0.0/0 > /tmp/sec_grp.log - fi - aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 443 --cidr 0.0.0.0/0 >> /tmp/sec_grp.log - report_status "$?" "creating security group rules" - - # Start provisioning - - echo "Creating VM at region $REGION, may take a few minutes." - - aws ec2 run-instances --region $REGION --image-id $AMI_IMAGE --count 1 --instance-type $EC2_TYPE --key-name $KEY_PAIR --security-group-ids $sg_id > vm_create.json - report_status "$?" "creating VM" - instance_id=$(jq -r .Instances[0].InstanceId vm_create.json) - - aws ec2 wait instance-status-ok --instance-ids $instance_id - aws ec2 describe-instances --instance-ids $instance_id > vm_result.json - - IP_ADDRESS=$(jq -r .Reservations[0].Instances[0].PublicIpAddress vm_result.json) - - echo "VM created with IP address: ${IP_ADDRESS}" - - echo "Installing docker engine in $VM_NAME, may take a few minutes." - DEST_SITE=${ADMIN_USERNAME}@${IP_ADDRESS} - scripts=$(cat << 'EOF' - sudo apt-get update && \ - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y ca-certificates curl gnupg lsb-release && \ - sudo mkdir -p /etc/apt/keyrings && \ - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg && \ - echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ - $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \ - sudo apt-get update && \ - sudo DEBIAN_FRONTEND=noninteractive apt-get install -y docker-ce docker-ce-cli containerd.io - EOF - ) - ssh -t -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} "$scripts" > /tmp/docker_engine.log - report_status "$?" "installing docker engine" - ssh -t -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} "sudo usermod -aG docker $ADMIN_USERNAME && exit" >> /tmp/docker_engine.log - report_status "$?" "installing docker engine" - - echo "Installing nvflare in $VM_NAME, may take a few minutes." - ssh -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ - "export PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin && \ - wget -q https://bootstrap.pypa.io/get-pip.py && python3 get-pip.py && \ - python3 -m pip install {~~NVFLARE~~} && \ - mkdir -p ./cert && \ - exit" > /tmp/nvflare.json - report_status "$?" "installing nvflare" - - echo "Checking if certificate (web.crt) and private key (web.key) are available" - if [[ -f "web.crt" && -f "web.key" ]]; then - CERT_FOLDER=${DEST_SITE}:${DEST_FOLDER}/cert - echo "Cert folder is ${CERT_FOLDER}" - scp -i $KEY_FILE -r -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null web.{crt,key} $CERT_FOLDER - report_status "$?" "copying cert/key to VM ${CERT_FOLDER} folder" - secure=true - else - echo "No web.crt and web.key found" - secure=false - fi - - echo "Starting dashboard" - ssh -i $KEY_FILE -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${DEST_SITE} \ - "export PATH=/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin && \ - python3 -m nvflare.dashboard.cli --start -f ${DEST_FOLDER} --cred ${credential} {~~START_OPT~~}" > /tmp/dashboard.json - - echo "Dashboard url is running at IP address ${IP_ADDRESS}, listening to port 443." - if [ "$secure" = true ] - then - echo "URL is https://${IP_ADDRESS}" - else - echo "URL is http://${IP_ADDRESS}:443" - fi - echo "Note: you may need to configure DNS server with your DNS hostname and the above IP address." - echo "Project admin credential (username:password) is ${credential} ." - echo "To terminate the EC2 instance, run the following command." - echo "aws ec2 terminate-instances --instance-ids ${instance_id}" - echo "Other resources provisioned" - echo "security group: ${SECURITY_GROUP}" - echo "key pair: ${KEY_PAIR}" diff --git a/nvflare/lighter/impl/static_file.py b/nvflare/lighter/impl/static_file.py index 21ef4c8f04..a024c84a43 100644 --- a/nvflare/lighter/impl/static_file.py +++ b/nvflare/lighter/impl/static_file.py @@ -18,8 +18,8 @@ import yaml +from nvflare.lighter import utils from nvflare.lighter.spec import Builder -from nvflare.lighter.utils import sh_replace class StaticFileBuilder(Builder): @@ -61,13 +61,6 @@ def __init__( self.snapshot_persistor = snapshot_persistor self.components = components - def _write(self, file_full_path, content, mode, exe=False): - mode = mode + "w" - with open(file_full_path, mode) as f: - f.write(content) - if exe: - os.chmod(file_full_path, 0o755) - def get_server_name(self, server): return server.name @@ -76,7 +69,7 @@ def get_overseer_name(self, overseer): def _build_overseer(self, overseer, ctx): dest_dir = self.get_kit_dir(overseer, ctx) - self._write( + utils._write( os.path.join(dest_dir, "start.sh"), self.template["start_svr_sh"], "t", @@ -95,7 +88,7 @@ def _build_overseer(self, overseer, ctx): privilege_dict[role].append(admin.subject) else: privilege_dict[role] = [admin.subject] - self._write( + utils._write( os.path.join(dest_dir, "privilege.yml"), yaml.dump(privilege_dict, Dumper=yaml.Dumper), "t", @@ -103,19 +96,19 @@ def _build_overseer(self, overseer, ctx): ) if self.docker_image: - self._write( + utils._write( os.path.join(dest_dir, "docker.sh"), - sh_replace(self.template["docker_svr_sh"], replacement_dict), + utils.sh_replace(self.template["docker_svr_sh"], replacement_dict), "t", exe=True, ) - self._write( + utils._write( os.path.join(dest_dir, "gunicorn.conf.py"), - sh_replace(self.template["gunicorn_conf_py"], replacement_dict), + utils.sh_replace(self.template["gunicorn_conf_py"], replacement_dict), "t", exe=False, ) - self._write( + utils._write( os.path.join(dest_dir, "start.sh"), self.template["start_ovsr_sh"], "t", @@ -140,11 +133,6 @@ def _build_server(self, server, ctx): server_0["service"]["scheme"] = self.scheme server_0["admin_host"] = self.get_server_name(server) server_0["admin_port"] = admin_port - # if self.download_job_url: - # server_0["download_job_url"] = self.download_job_url - # config["enable_byoc"] = server.enable_byoc - # if self.app_validator: - # config["app_validator"] = {"path": self.app_validator} if self.overseer_agent: overseer_agent = copy.deepcopy(self.overseer_agent) if overseer_agent.get("overseer_exists", True): @@ -158,46 +146,36 @@ def _build_server(self, server, ctx): } overseer_agent.pop("overseer_exists", None) config["overseer_agent"] = overseer_agent - # if self.snapshot_persistor: - # config["snapshot_persistor"] = self.snapshot_persistor - # components = server.props.get("components", []) - # config["components"] = list() - # for comp in components: - # temp_dict = {"id": comp} - # temp_dict.update(components[comp]) - # config["components"].append(temp_dict) - # provisioned_client_list = list() - # for client in self.project.get_participants_by_type("client", first_only=False): - # provisioned_client_list.append(client.name) - # config["provisioned_client_list"] = provisioned_client_list - self._write(os.path.join(dest_dir, "fed_server.json"), json.dumps(config, indent=2), "t") + utils._write(os.path.join(dest_dir, "fed_server.json"), json.dumps(config, indent=2), "t") replacement_dict = { "admin_port": admin_port, "fed_learn_port": fed_learn_port, "config_folder": self.config_folder, "docker_image": self.docker_image, "org_name": server.org, + "type": "server", + "cln_uid": "", } if self.docker_image: - self._write( + utils._write( os.path.join(dest_dir, "docker.sh"), - sh_replace(self.template["docker_svr_sh"], replacement_dict), + utils.sh_replace(self.template["docker_svr_sh"], replacement_dict), "t", exe=True, ) - self._write( + utils._write( os.path.join(dest_dir, "start.sh"), self.template["start_svr_sh"], "t", exe=True, ) - self._write( + utils._write( os.path.join(dest_dir, "sub_start.sh"), - sh_replace(self.template["sub_start_svr_sh"], replacement_dict), + utils.sh_replace(self.template["sub_start_sh"], replacement_dict), "t", exe=True, ) - self._write( + utils._write( os.path.join(dest_dir, "stop_fl.sh"), self.template["stop_fl_sh"], "t", @@ -205,29 +183,29 @@ def _build_server(self, server, ctx): ) # local folder creation dest_dir = self.get_local_dir(server, ctx) - self._write( + utils._write( os.path.join(dest_dir, "log.config.default"), self.template["log_config"], "t", ) - self._write( + utils._write( os.path.join(dest_dir, "resources.json.default"), self.template["local_server_resources"], "t", ) - self._write( + utils._write( os.path.join(dest_dir, "privacy.json.sample"), self.template["sample_privacy"], "t", ) - self._write( + utils._write( os.path.join(dest_dir, "authorization.json.default"), self.template["default_authz"], "t", ) # workspace folder file - self._write( + utils._write( os.path.join(self.get_ws_dir(server, ctx), "readme.txt"), self.template["readme_fs"], "t", @@ -247,6 +225,8 @@ def _build_client(self, client, ctx): "config_folder": self.config_folder, "docker_image": self.docker_image, "org_name": client.org, + "type": "client", + "cln_uid": f"uid={client.subject}", } if self.overseer_agent: overseer_agent = copy.deepcopy(self.overseer_agent) @@ -266,27 +246,27 @@ def _build_client(self, client, ctx): # temp_dict.update(components[comp]) # config["components"].append(temp_dict) - self._write(os.path.join(dest_dir, "fed_client.json"), json.dumps(config, indent=2), "t") + utils._write(os.path.join(dest_dir, "fed_client.json"), json.dumps(config, indent=2), "t") if self.docker_image: - self._write( + utils._write( os.path.join(dest_dir, "docker.sh"), - sh_replace(self.template["docker_cln_sh"], replacement_dict), + utils.sh_replace(self.template["docker_cln_sh"], replacement_dict), "t", exe=True, ) - self._write( + utils._write( os.path.join(dest_dir, "start.sh"), self.template["start_cln_sh"], "t", exe=True, ) - self._write( + utils._write( os.path.join(dest_dir, "sub_start.sh"), - sh_replace(self.template["sub_start_cln_sh"], replacement_dict), + utils.sh_replace(self.template["sub_start_sh"], replacement_dict), "t", exe=True, ) - self._write( + utils._write( os.path.join(dest_dir, "stop_fl.sh"), self.template["stop_fl_sh"], "t", @@ -294,29 +274,29 @@ def _build_client(self, client, ctx): ) # local folder creation dest_dir = self.get_local_dir(client, ctx) - self._write( + utils._write( os.path.join(dest_dir, "log.config.default"), self.template["log_config"], "t", ) - self._write( + utils._write( os.path.join(dest_dir, "resources.json.default"), self.template["local_client_resources"], "t", ) - self._write( + utils._write( os.path.join(dest_dir, "privacy.json.sample"), self.template["sample_privacy"], "t", ) - self._write( + utils._write( os.path.join(dest_dir, "authorization.json.default"), self.template["default_authz"], "t", ) # workspace folder file - self._write( + utils._write( os.path.join(self.get_ws_dir(client, ctx), "readme.txt"), self.template["readme_fc"], "t", @@ -335,21 +315,21 @@ def _build_admin(self, admin, ctx): config = self.prepare_admin_config(admin, ctx) - self._write(os.path.join(dest_dir, "fed_admin.json"), json.dumps(config, indent=2), "t") + utils._write(os.path.join(dest_dir, "fed_admin.json"), json.dumps(config, indent=2), "t") if self.docker_image: - self._write( + utils._write( os.path.join(dest_dir, "docker.sh"), - sh_replace(self.template["docker_adm_sh"], replacement_dict), + utils.sh_replace(self.template["docker_adm_sh"], replacement_dict), "t", exe=True, ) - self._write( + utils._write( os.path.join(dest_dir, "fl_admin.sh"), - sh_replace(self.template["fl_admin_sh"], replacement_dict), + utils.sh_replace(self.template["fl_admin_sh"], replacement_dict), "t", exe=True, ) - self._write( + utils._write( os.path.join(dest_dir, "readme.txt"), self.template["readme_am"], "t", diff --git a/nvflare/lighter/impl/template.py b/nvflare/lighter/impl/template.py index a85277ad11..e3a19e8261 100644 --- a/nvflare/lighter/impl/template.py +++ b/nvflare/lighter/impl/template.py @@ -26,6 +26,8 @@ class TemplateBuilder(Builder): def initialize(self, ctx): resource_dir = self.get_resources_dir(ctx) - template_file = ctx.get("template_file") - template = load_yaml(os.path.join(resource_dir, template_file)) + template_files = ctx.get("template_files") + template = dict() + for tplt_file in template_files: + template.update(load_yaml(os.path.join(resource_dir, tplt_file))) ctx["template"] = template diff --git a/nvflare/lighter/impl/workspace.py b/nvflare/lighter/impl/workspace.py index 4926b20629..6b203227df 100644 --- a/nvflare/lighter/impl/workspace.py +++ b/nvflare/lighter/impl/workspace.py @@ -43,9 +43,9 @@ def __init__(self, template_file): wip/ <--- this is only used during runtime, and will be removed when the provision command exits Args: - template_file: name of template file containing scripts and configs to put into startup folders + template_file: name(s) of template file(s) containing scripts and configs to put into startup folders """ - self.template_file = template_file + self.template_files = template_file def _make_dir(self, dirs): for dir in dirs: @@ -61,10 +61,15 @@ def initialize(self, ctx): if stage > last: last = stage ctx["last_prod_stage"] = last - template_file_full_path = os.path.join(self.get_resources_dir(ctx), self.template_file) - file_path = pathlib.Path(__file__).parent.absolute() - shutil.copyfile(os.path.join(file_path, self.template_file), template_file_full_path) - ctx["template_file"] = self.template_file + if not isinstance(self.template_files, list): + self.template_files = [self.template_files] + tplt_file_list = [] + for tplt_file in self.template_files: + tplt_file_full_path = os.path.join(self.get_resources_dir(ctx), tplt_file) + file_path = pathlib.Path(__file__).parent.absolute() + shutil.copyfile(os.path.join(file_path, tplt_file), tplt_file_full_path) + tplt_file_list.append(tplt_file) + ctx["template_files"] = tplt_file_list def build(self, project: Project, ctx: dict): dirs = [self.get_kit_dir(p, ctx) for p in project.participants] diff --git a/nvflare/lighter/tplt_utils.py b/nvflare/lighter/tplt_utils.py index e0ebf5aec9..590052a5cd 100644 --- a/nvflare/lighter/tplt_utils.py +++ b/nvflare/lighter/tplt_utils.py @@ -13,9 +13,81 @@ # limitations under the License. +from . import utils + + class Template: def __init__(self, template): self.template = template + self.supported_csps = ("azure", "aws") def get_cloud_script_header(self): return self.template.get("cloud_script_header") + + def get_azure_server_start_sh(self, entity): + tmp = self.get_cloud_script_header() + self.get_azure_start_svr_header_sh() + self.get_azure_start_common_sh() + script = utils.sh_replace( + tmp, + { + "type": "server", + "docker_network": "--network host", + "cln_uid": "", + "server_name": entity.name, + "ORG": "", + }, + ) + return script + + def get_aws_server_start_sh(self, entity): + tmp = self.get_cloud_script_header() + self.template.get("aws_start_sh") + script = utils.sh_replace( + tmp, + { + "type": "server", + "inbound_rule": "aws ec2 authorize-security-group-ingress --group-id $sg_id --protocol tcp --port 8002-8003 --cidr 0.0.0.0/0 >> /tmp/sec_grp.log", + "cln_uid": "", + "server_name": entity.name, + "ORG": "", + }, + ) + return script + + def get_azure_client_start_sh(self, entity): + tmp = self.get_cloud_script_header() + self.get_azure_start_cln_header_sh() + self.get_azure_start_common_sh() + script = utils.sh_replace( + tmp, + {"type": "client", "docker_network": "", "cln_uid": f"uid={entity.name}", "ORG": entity.org}, + ) + return script + + def get_aws_client_start_sh(self, entity): + tmp = self.get_cloud_script_header() + self.template.get("aws_start_sh") + script = utils.sh_replace( + tmp, {"type": "client", "inbound_rule": "", "cln_uid": f"uid={entity.name}", "ORG": entity.org} + ) + return script + + def get_azure_start_svr_header_sh(self): + return self.template.get("azure_start_svr_header_sh") + + def get_azure_start_cln_header_sh(self): + return self.template.get("azure_start_cln_header_sh") + + def get_azure_start_common_sh(self): + return self.template.get("azure_start_common_sh") + + def get_sub_start_sh(self): + return self.template.get("sub_start_sh") + + def get_azure_svr_sh(self): + return self.get_cloud_script_header() + self.get_azure_start_svr_header_sh() + self.get_azure_start_common_sh() + + def get_azure_cln_sh(self): + return self.get_cloud_script_header() + self.get_azure_start_cln_header_sh() + self.get_azure_start_common_sh() + + def get_start_sh(self, csp, type, entity): + try: + func = getattr(self, f"get_{csp}_{type}_start_sh") + return func(entity) + except AttributeError: + return "" diff --git a/nvflare/lighter/utils.py b/nvflare/lighter/utils.py index fa202b480a..e7836537d8 100644 --- a/nvflare/lighter/utils.py +++ b/nvflare/lighter/utils.py @@ -224,3 +224,75 @@ def update_storage_locations( json_object = json.dumps(resources, indent=4) with open(target_resource, "w") as outfile: outfile.write(json_object) + + +def _write(file_full_path, content, mode, exe=False): + mode = mode + "w" + with open(file_full_path, mode) as f: + f.write(content) + if exe: + os.chmod(file_full_path, 0o755) + + +def _write_common(type, dest_dir, template, tplt, replacement_dict, config): + mapping = {"server": "svr", "client": "cln"} + _write(os.path.join(dest_dir, f"fed_{type}.json"), json.dumps(config, indent=2), "t") + _write( + os.path.join(dest_dir, "docker.sh"), + sh_replace(template[f"docker_{mapping[type]}_sh"], replacement_dict), + "t", + exe=True, + ) + _write( + os.path.join(dest_dir, "start.sh"), + sh_replace(template[f"start_{mapping[type]}_sh"], replacement_dict), + "t", + exe=True, + ) + _write( + os.path.join(dest_dir, "sub_start.sh"), + sh_replace(tplt.get_sub_start_sh(), replacement_dict), + "t", + exe=True, + ) + _write( + os.path.join(dest_dir, "stop_fl.sh"), + template["stop_fl_sh"], + "t", + exe=True, + ) + + +def _write_local(type, dest_dir, template, capacity=""): + _write( + os.path.join(dest_dir, "log.config.default"), + template["log_config"], + "t", + ) + _write( + os.path.join(dest_dir, "privacy.json.sample"), + template["sample_privacy"], + "t", + ) + _write( + os.path.join(dest_dir, "authorization.json.default"), + template["default_authz"], + "t", + ) + resources = json.loads(template["local_client_resources"]) + if type == "client": + for component in resources["components"]: + if "nvflare.app_common.resource_managers.gpu_resource_manager.GPUResourceManager" == component["path"]: + component["args"] = json.loads(capacity) + break + _write( + os.path.join(dest_dir, "resources.json.default"), + json.dumps(resources, indent=2), + "t", + ) + + +def _write_pki(type, dest_dir, cert_pair, root_cert): + _write(os.path.join(dest_dir, f"{type}.crt"), cert_pair.ser_cert, "b", exe=False) + _write(os.path.join(dest_dir, f"{type}.key"), cert_pair.ser_pri_key, "b", exe=False) + _write(os.path.join(dest_dir, "rootCA.pem"), root_cert, "b", exe=False) From 9c3cf1836f4b45612561f284279f98e3cad41412 Mon Sep 17 00:00:00 2001 From: Zhihong Zhang <100308595+nvidianz@users.noreply.github.com> Date: Tue, 23 Jan 2024 00:54:14 -0500 Subject: [PATCH 10/39] Added debug headers for all message route in CoreCell (#2301) --- nvflare/fuel/f3/cellnet/core_cell.py | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/nvflare/fuel/f3/cellnet/core_cell.py b/nvflare/fuel/f3/cellnet/core_cell.py index 57aeb7a9eb..9ca89a85e6 100644 --- a/nvflare/fuel/f3/cellnet/core_cell.py +++ b/nvflare/fuel/f3/cellnet/core_cell.py @@ -1159,7 +1159,18 @@ def _send_target_messages( if ep: reachable_targets[t] = ep else: - self.log_error(f"cannot send to '{t}': {err}", tm.message) + msg = Message(headers=copy.copy(tm.message.headers), payload=tm.message.payload) + msg.add_headers( + { + MessageHeaderKey.CHANNEL: tm.channel, + MessageHeaderKey.TOPIC: tm.topic, + MessageHeaderKey.FROM_CELL: self.my_info.fqcn, + MessageHeaderKey.TO_CELL: t, + MessageHeaderKey.ORIGIN: self.my_info.fqcn, + MessageHeaderKey.DESTINATION: t, + } + ) + self.log_error(f"cannot send to '{t}': {err}", msg) send_errs[t] = err for t, ep in reachable_targets.items(): @@ -1170,12 +1181,12 @@ def _send_target_messages( { MessageHeaderKey.CHANNEL: tm.channel, MessageHeaderKey.TOPIC: tm.topic, - MessageHeaderKey.ORIGIN: self.my_info.fqcn, MessageHeaderKey.FROM_CELL: self.my_info.fqcn, + MessageHeaderKey.TO_CELL: ep.name, + MessageHeaderKey.ORIGIN: self.my_info.fqcn, + MessageHeaderKey.DESTINATION: t, MessageHeaderKey.MSG_TYPE: MessageType.REQ, MessageHeaderKey.ROUTE: [(self.my_info.fqcn, time.time())], - MessageHeaderKey.DESTINATION: t, - MessageHeaderKey.TO_CELL: ep.name, } ) @@ -1506,11 +1517,12 @@ def send_reply(self, reply: Message, to_cell: str, for_req_ids: List[str], secur reply.add_headers( { MessageHeaderKey.FROM_CELL: self.my_info.fqcn, + MessageHeaderKey.TO_CELL: to_cell, MessageHeaderKey.ORIGIN: self.my_info.fqcn, - MessageHeaderKey.ROUTE: [(self.my_info.fqcn, time.time())], MessageHeaderKey.DESTINATION: to_cell, MessageHeaderKey.REQ_ID: for_req_ids, MessageHeaderKey.MSG_TYPE: MessageType.REPLY, + MessageHeaderKey.ROUTE: [(self.my_info.fqcn, time.time())], MessageHeaderKey.SECURE: secure, MessageHeaderKey.OPTIONAL: optional, } @@ -1926,9 +1938,9 @@ def _process_received_msg(self, endpoint: Endpoint, connection: Connection, mess MessageHeaderKey.CHANNEL: channel, MessageHeaderKey.TOPIC: topic, MessageHeaderKey.FROM_CELL: self.my_info.fqcn, + MessageHeaderKey.TO_CELL: endpoint.name, MessageHeaderKey.ORIGIN: self.my_info.fqcn, MessageHeaderKey.DESTINATION: origin, - MessageHeaderKey.TO_CELL: endpoint.name, MessageHeaderKey.REQ_ID: req_id, MessageHeaderKey.MSG_TYPE: MessageType.REPLY, MessageHeaderKey.ROUTE: [(self.my_info.fqcn, time.time())], From d88fbdd76bf1ae22a866feb0f78d2d0e7e50273e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Tue, 23 Jan 2024 09:24:18 -0800 Subject: [PATCH 11/39] Update execution api documentation and docstrings (#2305) --- docs/getting_started.rst | 4 +- docs/index.rst | 4 +- docs/programming_guide.rst | 2 +- .../controllers/controllers.rst | 10 +- .../controllers/initialize_global_weights.rst | 2 +- .../scatter_and_gather_workflow.rst | 2 +- docs/programming_guide/execution_api_type.rst | 89 +++++ .../3rd_party_integration.rst | 293 +++++++++------ .../execution_api_type/client_api.rst | 195 ++++++++++ .../executor.rst | 6 +- .../model_learner.rst | 0 docs/programming_guide/fl_clients.rst | 52 --- .../fl_clients/client_api.rst | 349 ------------------ docs/programming_guide/fl_model.rst | 17 + docs/programming_guide/resources/te.py | 9 - .../workflows_and_controllers.rst | 13 +- docs/release_notes/flare_230.rst | 4 +- docs/release_notes/flare_240.rst | 20 +- .../3rd_party_integration_diagram.png | Bin .../resources/fed_sag_round.png | Bin .../init_weights_1_config_fed_server.json | 0 .../task_execution_decision_chart.png | Bin 0 -> 73494 bytes docs/user_guide.rst | 10 +- docs/user_guide/nvflare_cli.rst | 7 +- .../step-by-step/cifar10/sag/sag.ipynb | 6 +- .../executors/client_api_launcher_executor.py | 8 +- .../app_common/executors/task_exchanger.py | 31 +- nvflare/app_opt/lightning/api.py | 38 +- nvflare/client/api.py | 183 ++++++++- nvflare/client/config.py | 107 +++--- nvflare/client/decorator.py | 36 ++ nvflare/client/flare_agent.py | 89 ++++- nvflare/client/model_registry.py | 19 +- nvflare/client/task_registry.py | 49 ++- nvflare/client/tracking.py | 24 +- nvflare/fuel/utils/pipe/cell_pipe.py | 12 +- nvflare/fuel/utils/pipe/pipe_handler.py | 17 +- 37 files changed, 1031 insertions(+), 676 deletions(-) create mode 100644 docs/programming_guide/execution_api_type.rst rename docs/programming_guide/{fl_clients => execution_api_type}/3rd_party_integration.rst (52%) create mode 100644 docs/programming_guide/execution_api_type/client_api.rst rename docs/programming_guide/{fl_clients => execution_api_type}/executor.rst (92%) rename docs/programming_guide/{fl_clients => execution_api_type}/model_learner.rst (100%) delete mode 100644 docs/programming_guide/fl_clients.rst delete mode 100644 docs/programming_guide/fl_clients/client_api.rst create mode 100644 docs/programming_guide/fl_model.rst delete mode 100644 docs/programming_guide/resources/te.py rename docs/{programming_guide => }/resources/3rd_party_integration_diagram.png (100%) rename docs/{programming_guide => }/resources/fed_sag_round.png (100%) rename docs/{programming_guide => }/resources/init_weights_1_config_fed_server.json (100%) create mode 100644 docs/resources/task_execution_decision_chart.png diff --git a/docs/getting_started.rst b/docs/getting_started.rst index d5bb9cb8ab..6f896b17b4 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -22,14 +22,14 @@ Clone NVFLARE repo to get examples, switch main branch (latest stable branch) $ git clone https://github.com/NVIDIA/NVFlare.git $ cd NVFlare - $ git switch main + $ git switch 2.4 Note on branches: * The `main `_ branch is the default (unstable) development branch -* The 2.0, 2.1, 2.2, and 2.3 etc. branches are the branches for each major release and minor patches +* The 2.1, 2.2, 2.3, and 2.4 etc. branches are the branches for each major release and minor patches Quick Start with Simulator diff --git a/docs/index.rst b/docs/index.rst index 78ea857358..d012124f12 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -21,7 +21,7 @@ NVIDIA FLARE glossary NVIDIA FLARE (NVIDIA Federated Learning Application Runtime Environment) is a domain-agnostic, open-source, extensible SDK that allows -researchers and data scientists to adaptexisting ML/DL workflows (PyTorch, RAPIDS, Nemo, TensorFlow) to a federated paradigm; and enables +researchers and data scientists to adapt existing ML/DL workflows (PyTorch, RAPIDS, Nemo, TensorFlow) to a federated paradigm; and enables platform developers to build a secure, privacy preserving offering for a distributed multi-party collaboration. NVIDIA FLARE is built on a componentized architecture that gives you the flexibility to take federated learning workloads from research @@ -34,7 +34,7 @@ and simulation to real-world production deployment. Some of the key components - **Management tools** for secure provisioning and deployment, orchestration, and management - **Specification-based API** for extensibility -Learn more in the :ref:`FLARE Overview `, :ref:`Key Features `, :ref:`What's New `, and the +Learn more in the :ref:`FLARE Overview `, :ref:`What's New `, and the :ref:`User Guide ` and :ref:`Programming Guide `. Getting Started diff --git a/docs/programming_guide.rst b/docs/programming_guide.rst index b181cbe7c9..835839245f 100644 --- a/docs/programming_guide.rst +++ b/docs/programming_guide.rst @@ -36,7 +36,7 @@ Please refer to :ref:`application` for more details. :maxdepth: 1 programming_guide/workflows_and_controllers - programming_guide/fl_clients + programming_guide/execution_api_type programming_guide/shareable programming_guide/data_exchange_object programming_guide/fl_context diff --git a/docs/programming_guide/controllers/controllers.rst b/docs/programming_guide/controllers/controllers.rst index dd611b3631..cf4a8f4368 100644 --- a/docs/programming_guide/controllers/controllers.rst +++ b/docs/programming_guide/controllers/controllers.rst @@ -73,7 +73,9 @@ The Controller's Task Manager manages the task's lifecycle: .. note:: - In NVIDIA FLARE 2.0, the underlying communication is by gRPC: the client always initiates communication by sending - a request to the server and a receiving response. When we say "server sends task to the client", it is only - conceptual. With gRPC, the client sends the "ask for next task" request to the server, and the server responds with - the task data. + In NVIDIA FLARE, the underlying communication is facilitated through gRPC: + the client always initiates communication by sending a request to the server and receiving a response. + When referring to the scenario where the "server sends a task to the client," + it is important to note that this is a conceptual representation. + In reality, with gRPC, the client initiates the interaction by sending a "request for the next task" to the server, + and the server responds by providing the task data. diff --git a/docs/programming_guide/controllers/initialize_global_weights.rst b/docs/programming_guide/controllers/initialize_global_weights.rst index aed6b035d0..6634d3133e 100644 --- a/docs/programming_guide/controllers/initialize_global_weights.rst +++ b/docs/programming_guide/controllers/initialize_global_weights.rst @@ -26,7 +26,7 @@ Two changes are needed: The updated file should look like the following: -.. literalinclude:: ../resources/init_weights_1_config_fed_server.json +.. literalinclude:: ../../resources/init_weights_1_config_fed_server.json :language: json diff --git a/docs/programming_guide/controllers/scatter_and_gather_workflow.rst b/docs/programming_guide/controllers/scatter_and_gather_workflow.rst index ad5b1d9507..44c9d232a8 100644 --- a/docs/programming_guide/controllers/scatter_and_gather_workflow.rst +++ b/docs/programming_guide/controllers/scatter_and_gather_workflow.rst @@ -7,7 +7,7 @@ of NVIDIA FLARE with a Server aggregating results from Clients that have produce At the core, the control_flow of :class:`nvflare.app_common.workflows.scatter_and_gather.ScatterAndGather` is a for loop: -.. image:: ../resources/fed_sag_round.png +.. image:: ../../resources/fed_sag_round.png :height: 400px Trainer diff --git a/docs/programming_guide/execution_api_type.rst b/docs/programming_guide/execution_api_type.rst new file mode 100644 index 0000000000..1095368afc --- /dev/null +++ b/docs/programming_guide/execution_api_type.rst @@ -0,0 +1,89 @@ +.. _execution_api_type: + +################## +Execution API Type +################## + +In the FLARE system, a federated learning algorithm is defined in a Job format +(for details, please refer to :ref:`job`). +A Job consists of multiple "workflows" and "executors." + +The simplified job execution flow is as follows: + +- The workflow schedules a task for the FL clients. +- Each FL client performs the received task and sends the result back. +- The workflow receives the results and determines if it is done. +- If it is not done, it schedules a new task +- If it is done, it proceeds to the next workflow in the Job. + +Users need to adapt their local training logic into FLARE's task execution +abstractions to make their training federated. + +We offer various levels of abstraction for writing task execution code, +catering to use cases that span from complete customizability to easy user adaptation. + +Below is a general overview of the key ideas and use cases for each type: + +**Client API** + +The :ref:`client_api` provides the most straightforward way to write FL code, +and can easily be used to convert centralized code with minimal code changes. +The Client API uses the :class:`FLModel` +object for data transfer and supports common tasks such as train, validate, and submit_model. +Additionally, options for using decorators or PyTorch Lightning are also available. + +We recommend users start with the Client API, and to consider the other types for more specific cases as required. + +**ModelLearner** + +The :ref:`model_learner` is designed to simplify writing learning logic by +minimizing FLARE-specific concepts. +The :class:`ModelLearner` +defines familiar learning functions for training and validation, +and uses the :class:`FLModel` +object for transferring learning information. +The ModelLearner also contains several convenient capabilities, +such as lifecycle and logging information. + +The ModelLearner is best used when working with standard machine learning code +that can fit well into the train and validate methods and can be easily adapted +to the ModelLearner subclass and method structure. + +**Executor** + +:ref:`executor` is the most flexible for defining custom logic and tasks, +as with a custom executor and controller, any form of computation can be performed. +However, Executors must deal directly with FLARE-specific communication concepts +such as :class:`Shareable`, :class:`DXO`, +and :class:`FLContext`. +As a result, many higher-level APIs are built on top of Executors in order to +abstract these concepts away for easier user adaptation. + +Overall, writing an Executor is most useful when implementing tasks and logic +that do not fit within the structure of higher-level APIs or other predefined Executors. + +**3rd-Party System Integration** + +There are cases where users have a pre-existing ML/DL training system +infrastructure that cannot be easily adapted to the FLARE client. + +The :ref:`3rd_party_integration` pattern allows for a seamless integration +between the FLARE system and a third-party external training system. + +With the use of the :mod:`FlareAgent ` and +:mod:`TaskExchanger `, +users can easily enable any 3rd-party system to receive tasks and submit results back to the server. + +Please use the following chart to decide which abstraction to use: + +.. image:: ../resources/task_execution_decision_chart.png + +For more details about each type, refer to each page below. + +.. toctree:: + :maxdepth: 1 + + execution_api_type/3rd_party_integration + execution_api_type/client_api + execution_api_type/model_learner + execution_api_type/executor diff --git a/docs/programming_guide/fl_clients/3rd_party_integration.rst b/docs/programming_guide/execution_api_type/3rd_party_integration.rst similarity index 52% rename from docs/programming_guide/fl_clients/3rd_party_integration.rst rename to docs/programming_guide/execution_api_type/3rd_party_integration.rst index 4c83823f3b..bdb41ec555 100644 --- a/docs/programming_guide/fl_clients/3rd_party_integration.rst +++ b/docs/programming_guide/execution_api_type/3rd_party_integration.rst @@ -4,32 +4,149 @@ 3rd-Party System Integration ############################ -NVFLARE 2.4.0 supports 3rd-party external systems to integrate with FL clients. +NVFLARE supports a seamless integration between the FLARE system and a +third-party external training system. +This is especially useful with pre-existing ML/DL training system +infrastructure that cannot be easily adapted to the FLARE client. -The FL Client installs the :mod:`TaskExchanger` executor and -the 3rd-party system uses the :mod:`FlareAgent` to interact with the TaskExchanger to receive tasks, and submit results to the FLARE server. +The FL Client uses the :class:`TaskExchanger` +executor to receive tasks, and submit results to the FLARE server. +The 3rd-party system uses the :class:`FlareAgent` to +interact with the TaskExchanger to get tasks and submit results. This integration pattern is illustrated in the diagram below: -.. image:: ../resources/3rd_party_integration_diagram.png +.. image:: ../../resources/3rd_party_integration_diagram.png :height: 400px Requirements ============ - The key to enabling this integration is the "agent_id" that must be made known to both systems. - The FL client gets this information from the job's client_config.json, and the 3rd-party trainer gets this from its own launch process or via the :class:`Piper`. -- It is assumed that the customer already has a way to dynamically generate the "agent_id" for each job, and start its trainer process with this information. -- Each FL client must be able to open an address (host:port) to allow the trainer to connect to. Depending on where the trainer is running, the connection may or may not need to be in secure mode (TLS). + The FL client gets this information from the job's config_fed_client, and the + 3rd-party trainer gets this from its own launch process. +- It is assumed that the customer already has a way to dynamically generate the + "agent_id" for each job, and start its trainer process with this information. +- Each FL client must be able to open an address (host:port) to allow the trainer to connect to. + Depending on where the trainer is running, the connection may or may not need to be in secure mode (TLS). +- We will need to modify the "project.yml" for NVFlare provision system + and generate new package folders for each participating sites - The trainer must be a Python program that can integrate with the NVFLARE library. -- The trainer must be able to connect to the server, as well as the address that is dynamically opened by the FL client. +- The trainer must be able to connect to the server, as well as the address that + is dynamically opened by the FL client. Prepare the Trainer =================== -You need to modify your trainer code to integrate with the FlareAgent API. -This API provides simple `get_task()` and `submit_result()` methods to interact with the FL client (FL client). -The following is an example of this usage pattern. +Let's prepare the trainer code first, we will modify the "project.yml" in the +next section for project setup. + +You need to modify your trainer code to integrate with the :class:`FlareAgent` API. +This API provides simple ``get_task()`` and ``submit_result()`` methods to interact with the FL client. + +We will go through the steps one by one: + +1. Create Agent +--------------- + +The :class:`FlareAgent` is responsible +for interacting with the FL client to exchange task data. + +If using FLModel, :class:`FlareAgentWithFLModel` +subclasses FlareAgent and provides conversion from shareables to task using the FLModel data structure. + +If using CellPipe, a convenient class :class:`FlareAgentWithCellPipe` +can be used. + +Please refer to their API page for detailed explanations of each argument: + + - :class:`FlareAgent` + - :class:`FlareAgentWithFLModel` + - :class:`FlareAgentWithCellPipe` + +You can create the FlareAgent as the following code: + +.. code-block:: python + + from nvflare.client.flare_agent import FlareAgentWithCellPipe + + agent = FlareAgentWithCellPipe( + root_url="grpc://server:8002", + flare_site_name=args.site_name, + agent_id=args.agent_id, + workspace_dir=args.workspace, + secure_mode=True, + submit_result_timeout=2.0, + heartbeat_timeout=120.0, + ) + +2. Start Agent +-------------- + +After we create the agent, we need to start it. +We can call ``agent.start()`` to start the agent. +This call must be made before trying to get tasks. + +For example: + +.. code-block:: python + + agent.start() + +3. Process Tasks +---------------- + +The training is a continuous process of getting a task, executing the task, +and submitting the task result. + +Call ``agent.get_task()`` to get a Task object from the FL client. +This is a blocking call and returns only when a task is available. +If there are no more tasks available (i.e. end of the job), ``AgentClosed`` +exception will be raised, and signaling to end the training. + +The :class:`Task` object contains 3 pieces of +information: task_name, task_id, and data. +The task_name tells you what the task is (e.g. train). +The task_id is a UUID of the task instance. +The data contains model data to be trained on. + +Once the task is completed, the result can be submitted to the FL client by calling ``agent.submit_result()``. +A return code (``rc``) must be provided to indicate whether the task was executed successfully. +If the ``rc`` is not RC.OK, then the job will be aborted. + +For example: + +.. code-block:: python + + while True: + print("getting task ...") + try: + task = agent.get_task() + except AgentClosed: + print("agent closed - exit") + break + + print(f"got task: {task}") + rc, meta, result = train(task.data) # perform train task + submitted = agent.submit_result(TaskResult(data=result, meta=meta, return_code=rc)) + print(f"result submitted: {submitted}") + +4. Stop Agent +------------- + +At the end of the training, ``agent.stop()`` must be called to end the program gracefully. +If this call is missed, the program may not exit properly. + +.. code-block:: python + + agent.stop() + + +Putting Together +---------------- + +Now we learn all the necessary steps, we can put together into the following +example code of this usage pattern: .. code-block:: python @@ -54,6 +171,7 @@ The following is an example of this usage pattern. args = parser.parse_args() + # 1. create the agent agent = FlareAgentWithCellPipe( root_url="grpc://server:8002", flare_site_name=args.site_name, @@ -64,8 +182,10 @@ The following is an example of this usage pattern. heartbeat_timeout=120.0, ) + # 2. start the agent agent.start() + # 3. processing tasks while True: print("getting task ...") try: @@ -75,10 +195,11 @@ The following is an example of this usage pattern. break print(f"got task: {task}") - rc, meta, result = train(task.data) # peform train task + rc, meta, result = train(task.data) # perform train task submitted = agent.submit_result(TaskResult(data=result, meta=meta, return_code=rc)) print(f"result submitted: {submitted}") + # 4. stop the agent agent.stop() @@ -88,79 +209,34 @@ The following is an example of this usage pattern. if __name__ == "__main__": main() -Create the Agent ----------------- - -The :class:`FlareAgent` is responsible for interacting with the FL client to exchange task data takes the following parameters: - -- ``pipe`` - component id of pipe for communication -- ``read_interval`` - how often to read from pipe -- ``heartbeat_interval`` - how often to send heartbeat to peer -- ``heartbeat_timeout`` - max amount of time to allow missing heartbeats before treating peer as dead -- ``resend_interval`` - how often to resend a message when failing to send -- ``max_resends`` - max number of resends. None means no limit -- ``submit_result_timeout`` - when submitting task result, how long to wait for response from the FL client -- ``metric_pipe`` - component id of pipe for metrics -- ``task_channel_name`` - the channel name for tasks (defaults to PipeChannelName.TASK) -- ``metric_channel_name`` - the channel name for metrics (defaults to PipeChannelName.METRIC) -- ``close_pipe`` - whether pipe needs to be closed (FilePipe: False, CellPipe: True) - -If using FLModel, :class:`FlareAgentWithFLModel` subclasses FlareAgent and provides conversion from shareables to task using the FLModel data structure. - -If using CellPipe, then :class:`FlareAgentWithCellPipe` subclasses FlareAgent and takes the parameters: - -- ``agent_id`` - this is the ID of the agent dynamically generated by your launch system -- ``site_name`` - this is the name of the FL client provisioned for the project -- ``root_url`` - this is the URL of the server. -- ``secure_mode`` - whether the trainer/FL client communication will be in secure mode (SSL) -- ``workspace_dir`` - this is the local folder that contains the "startup" kit of the FL client site. The trainer system and the FL client must share the same "startup" content. - -Start the Agent ---------------- - -Call ``agent.start()`` to start the agent. This call must be made before trying to get tasks. - -Process Tasks -------------- - -The training is a continuous process of getting a task, executing the task, and submitting the task result. - -Call ``agent.get_task()`` to get a Task object from the FL client. This is a blocking call and returns only when a task is available. -If there are no more tasks available (i.e. end of the job), ``AgentClosed`` exception will be raised, and signaling to end the training. - -The :class:`Task` object contains 3 pieces of information: task_name, task_id, and data. -he task_name tells you what the task is (e.g. train). The task_id is a UUID of the task instance. -The data contains model data to be trained on. - -Once the task is completed, the result can be submitted to the FL client by calling ``agent.submit_result()``. -A return code (``rc``) must be provided to indicate whether the task was executed successfully. -If the ``rc`` is not RC.OK, then the job will be aborted. - -Stop Agent ----------- - -At the end of the training, ``agent.stop()`` must be called to end the program gracefully. -If this call is missed, the program may not exit properly. - Notes: - This pattern of (``start``, ``get_task``, ``submit_result``, and ``stop``) is strictly enforced. - If the pattern is not followed (e.g. ``get_task``, then ``get_task`` again without ``submit_result``), you will get a ``CallStateError`` exception. + If the pattern is not followed (e.g. ``get_task``, then ``get_task`` again without ``submit_result``), + you will get a ``CallStateError`` exception. - The only way to know that the job is ended is the ``AgentClosed`` exception from the ``get_task`` call. - This exception is raised when the FL client tells the agent that the job is done; or when the FL client is considered dead (missing heartbeats for the configured period of time). -- If your training algorithm runs into an unrecoverable error and wants to end the job, you should use a proper return code (e.g. ``RC.EXECUTION_EXCEPTION``). + This exception is raised when the FL client tells the agent that the job is done; + or when the FL client is considered dead (missing heartbeats for the configured period of time). +- If your training algorithm runs into an unrecoverable error and wants to end the job, + you should use a proper return code (e.g. ``RC.EXECUTION_EXCEPTION``). Project Setup ============= -The following steps show you how to properly set up your project and jobs. +After we prepare the trainer code we can follow the steps below to properly +set up the project and jobs. Step One - Provision -------------------- -The FL client will behave like both client and server for connecting from the perspective of the trainer. +The FL client site will behave like both client and server for connecting from the perspective of the trainer. This requires the client site to have two sets of TLS credentials. -Make sure to specify the "listening_host" for the client in the project.yml when provisioning the project: +Make sure to specify the "listening_host" for the client in the project.yml when provisioning the project. + +.. note:: + We assume you understand NVFlare provision, if not please read :ref:`provisioning`. + +An example looks like: .. code-block:: yaml @@ -180,17 +256,20 @@ Make sure to specify the "listening_host" for the client in the project.yml when org: nvidia listening_host: site_2.maglev.nvidia.com -Once the project is provisioned, check the "startup" kit generated for the clients. You should see the following files, among others: +Once the project is provisioned, check the "startup" kit generated for the clients. +You should see the following files, among others: client.crt, client.key, server.crt, server.key, rootCA.pem Note that the specified listening_port of a site must be accessible to the trainer of the site. -Step Two - Setup for adhoc direct connection between FL Client and Trainer +Step Two - Setup for Adhoc Direct Connection between FL Client and Trainer -------------------------------------------------------------------------- -FL client and the trainer can always talk to each other via the server, but it could be slow, especially if the server is located far away. -The enable adhoc direct connections between the FL client and the Trainer, configure the comm_config.json on the client site as follows: +FL client and the trainer can always talk to each other via the server, +but it could be slow, especially if the server is located far away. +The enable adhoc direct connections between the FL client and the trainer, +configure the comm_config.json on the client site as follows: .. code-block:: json @@ -210,16 +289,19 @@ This file must be placed into the site's "local" folder within its workspace. Pay attention to the following: -- For most cases, the "scheme" should be set to "tcp" to get the best performance. If "tcp" cannot be used, you can use "grpc". +- For most cases, the "scheme" should be set to "tcp" to get the best performance. + If "tcp" cannot be used, you can use "grpc". - In "resources": - - If FL client and the Trainer are within the same trusted network, you can set "secure" to false; otherwise set it to true; + - If FL client and the trainer are within the same trusted network, + you can set "secure" to false; otherwise set it to true. - The value of the "host" must match the "listening_host" value of the site used in provision. -Step Three - Prepare job configuration +Step Three - Prepare Job Configuration -------------------------------------- -For each job, configure the config_fed_client.json to use :mod:`TaskExchanger` as the executor. +For each job, configure the config_fed_client.json to use +:class:`TaskExchanger` as the executor. .. code-block:: json @@ -228,7 +310,7 @@ For each job, configure the config_fed_client.json to use :mod:`TaskExchanger` +are configured properly, and change the default values as needed. -- ``pipe_id`` - component id of pipe -- ``read_interval`` - how often to read from pipe -- ``heartbeat_interval`` - how often to send heartbeat to peer -- ``heartbeat_timeout`` - max amount of time to allow missing heartbeats before treating peer as dead -- ``resend_interval`` - how often to resend a message when failing to send -- ``max_resends`` - max number of resends. None means no limit -- ``peer_read_timeout`` - time to wait for peer to accept sent message -- ``task_wait_time`` - how long to wait for a task to complete. None means waiting forever -- ``result_poll_interval`` - how often to poll task result -- ``pipe_channel_name`` - the channel name for sending task requests +Please refer to the API page for a detailed explanation of each argument: +:class:`TaskExchanger` Step Four - Trainer Setup ------------------------- -The trainer program must have access to a local file system, and you must create a "workspace" folder. This workspace should be used for all jobs. +The trainer program must have access to a local file system, and you must create a "workspace" folder. +This workspace should be used for all jobs. Copy the "startup" folder of the provisioned site, and put it in the designated workspace folder. If needed, any additional config files required by the trainer can also be placed in the workspace folder. -Ensure to set the FlareAgent's "workspace_dir" to the workspace folder and that the correct "agent_id" value is passed to both the FL client and the training process. +Ensure to set the FlareAgent's "workspace_dir" to the workspace folder and +that the correct "agent_id" value is passed to both the FL client and the training process. Verification ============ -The FL client (TaskExchanger) and your trainer process (FlareAgent) do not have to be started at exactly the same time. Whichever is started first will wait for the other for ``heartbeat_timeout`` seconds. -Once they both are started and connected, you can verify they are directly connected using the Admin's cell commands. +The FL client (TaskExchanger) and your trainer process (FlareAgent) do not have +to be started at exactly the same time. +Whichever is started first will wait for the other for ``heartbeat_timeout`` seconds. +Once they both are started and connected, you can verify they are directly +connected using the Admin console's ``cells`` commands. -The following example shows two clients (red, blue) connected to their external trainers via the agent_id "ext_trainer_1": +The following example shows two clients (red, blue) connected to their external +trainers via the agent_id "ext_trainer_1": .. code-block:: shell @@ -292,7 +371,8 @@ The following example shows two clients (red, blue) connected to their external Total Cells: 8 Done [21695 usecs] 2023-10-16 19:28:37.523651 -The ``cells`` command lists all cells. Notice that the job 44c08365-e829-4bc1-a034-cda5a252fe73 is running on both "blue" and "red" clients. +The ``cells`` command lists all cells. +Notice that the job 44c08365-e829-4bc1-a034-cda5a252fe73 is running on both "blue" and "red" clients. Also notice that there are two corresponding ext_trainer cells (red-ext_trainer_1, and blue-ext_trainer1). .. code-block:: shell @@ -304,7 +384,8 @@ Also notice that there are two corresponding ext_trainer cells (red-ext_trainer_ Done [14526 usecs] 2023-10-16 19:28:44.407505 The ``peers`` command shows the cells directly connected to the specified cell. -Here you see that the blue-ext_trainer_1 is directly connected to two cells: the server and the FL client (blue.44c08365-e829-4bc1-a034-cda5a252fe73). +Here you see that the blue-ext_trainer_1 is directly connected to two cells: +the server and the FL client (blue.44c08365-e829-4bc1-a034-cda5a252fe73). .. code-block:: shell @@ -324,6 +405,8 @@ Here you see that the blue-ext_trainer_1 is directly connected to two cells: the } } -The ``conns`` command shows the connectors on the specified cell. Here you see that blue--ext_trainer_1 has two connectors: -one connects the server on ``grpc://server:8002``, and another connects to ``blue.44c08365-e829-4bc1-a034-cda5a252fe73 on stcp://nvclient:11947``. -Note that this port is opened by the FL client dynamically. +The ``conns`` command shows the connectors on the specified cell. +Here you see that blue--ext_trainer_1 has two connectors: +one connects the server on ``grpc://server:8002``, and another connects to +``blue.44c08365-e829-4bc1-a034-cda5a252fe73 on stcp://nvclient:11947``. +Note that this port (11947) is opened by the FL client dynamically. diff --git a/docs/programming_guide/execution_api_type/client_api.rst b/docs/programming_guide/execution_api_type/client_api.rst new file mode 100644 index 0000000000..0861a446e3 --- /dev/null +++ b/docs/programming_guide/execution_api_type/client_api.rst @@ -0,0 +1,195 @@ +.. _client_api: + +########## +Client API +########## + +The FLARE Client API provides an easy way for users to convert their centralized, +local training code into federated learning code with the following benefits: + +* Only requires a few lines of code changes, without the need to restructure the code or implement a new class +* Reduces the number of new FLARE specific concepts exposed to users +* Easy adaptation from existing local training code using different frameworks + (PyTorch, PyTorch Lightning, HuggingFace) + +Core concept +============ + +The general structure of the popular federated learning (FL) workflow, "FedAvg" is as follows: + +#. FL server initializes an initial model +#. For each round (global iteration): + + #. FL server sends the global model to clients + #. Each FL client starts with this global model and trains on their own data + #. Each FL client sends back their trained model + #. FL server aggregates all the models and produces a new global model + +On the client side, the training workflow is as follows: + +#. Receive the model from the FL server +#. Perform local training on the received global model and/or evaluate the + received global model for model selection +#. Send the new model back to the FL server + +To convert a centralized training code to federated learning, we need to +adapt the code to do the following steps: + +#. Obtain the required information from received :ref:`fl_model` +#. Run local training +#. Put the results in a new :ref:`fl_model` to be sent back + +For a general use case, there are three essential methods for the Client API: + +* ``init()``: Initializes NVFlare Client API environment. +* ``receive()``: Receives model from NVFlare side. +* ``send()``: Sends the model to NVFlare side. + +Users can use the Client API to change their centralized training code to +federated learning, for example: + +.. code-block:: python + + import nvflare.client as flare + + flare.init() # 1. Initializes NVFlare Client API environment. + input_model = flare.receive() # 2. Receives model from NVFlare side. + params = input_model.params # 3. Obtain the required information from received FLModel + + # original local training code begins + new_params = local_train(params) + # original local training code ends + + output_model = flare.FLModel(params=new_params) # 4. Put the results in a new FLModel + flare.send(output_model) # 5. Sends the model to NVFlare side. + +With 5 lines of code changes, we convert the centralized training code to +federated learning setting. + +After this, we can utilize the job templates and the :ref:`job_cli` +to generate a job so it can be run using :ref:`fl_simulator` +or submit to a deployed NVFlare system. + +Below is a table overview of key Client APIs. + +.. list-table:: Client API + :widths: 25 25 50 + :header-rows: 1 + + * - API + - Description + - API Doc Link + * - init + - Initializes NVFlare Client API environment. + - :func:`init` + * - receive + - Receives model from NVFlare side. + - :func:`receive` + * - send + - Sends the model to NVFlare side. + - :func:`send` + * - system_info + - Gets NVFlare system information. + - :func:`system_info` + * - get_job_id + - Gets job id. + - :func:`get_job_id` + * - get_site_name + - Gets site name. + - :func:`get_site_name` + * - is_running + - Returns whether the NVFlare system is up and running. + - :func:`is_running` + * - is_train + - Returns whether the current task is a training task. + - :func:`is_train` + * - is_evaluate + - Returns whether the current task is an evaluate task. + - :func:`is_evaluate` + * - is_submit_model + - Returns whether the current task is a submit_model task. + - :func:`is_submit_model` + +.. list-table:: Decorator APIs + :widths: 25 25 50 + :header-rows: 1 + + * - API + - Description + - API Doc Link + * - train + - A decorator to wraps the training logic. + - :func:`train` + * - evaluate + - A decorator to wraps the evaluate logic. + - :func:`evaluate` + +.. list-table:: Lightning APIs + :widths: 25 25 50 + :header-rows: 1 + + * - API + - Description + - API Doc Link + * - patch + - Patches the PyTorch Lightning Trainer for usage with FLARE. + - :func:`train` + +.. list-table:: Metrics Logger + :widths: 25 25 50 + :header-rows: 1 + + * - API + - Description + - API Doc Link + * - SummaryWriter + - SummaryWriter mimics the usage of Tensorboard's SummaryWriter. + - :class:`SummaryWriter` + * - WandBWriter + - WandBWriter mimics the usage of weights and biases. + - :class:`WandBWriter` + * - MLflowWriter + - MLflowWriter mimics the usage of MLflow. + - :class:`MLflowWriter` + +Please check Client API Module :mod:`nvflare.client.api` for more in-depth +information about all of the Client API functionalities. + +If you are using PyTorch Lightning in your training code, you can check the +Lightning API Module :mod:`nvflare.app_opt.lightning.api`. + + +Configuration +============= + +In the config_fed_client in the FLARE app, in order to launch the training +script we use the +:class:`SubprocessLauncher` component. +The defined ``script`` is invoked, and ``launch_once`` can be set to either +launch once for the whole job, or launch a process for each task received from the server. + +A corresponding :class:`LauncherExecutor` +is used as the executor to handle the tasks and perform the data exchange using the pipe. +For the Pipe component we provide implementations of :class:`FilePipe` +and :class:`CellPipe`. + +.. literalinclude:: ../../../job_templates/sag_pt/config_fed_client.conf + +For example configurations, take a look at the :github_nvflare_link:`job_templates ` +directory for templates using the launcher and Client API. + +.. note:: + In that case that the user does not need to launch the process and instead + has their own existing external training system, this would involve using + the :ref:`3rd_party_integration`, which is based on the same underlying mechanisms. + +Examples +======== + +For examples of using Client API with different frameworks, +please refer to :github_nvflare_link:`examples/hello-world/ml-to-fl `. + +For additional examples, also take a look at the +:github_nvflare_link:`step-by-step series ` +that use Client API to write the +:github_nvflare_link:`train script `. diff --git a/docs/programming_guide/fl_clients/executor.rst b/docs/programming_guide/execution_api_type/executor.rst similarity index 92% rename from docs/programming_guide/fl_clients/executor.rst rename to docs/programming_guide/execution_api_type/executor.rst index dac7bfba81..45d4bea8d3 100644 --- a/docs/programming_guide/fl_clients/executor.rst +++ b/docs/programming_guide/execution_api_type/executor.rst @@ -6,9 +6,9 @@ Executor .. image:: ../../resources/Executor.png :height: 300px -An :class:`Executor` in NVIDIA FLARE is a type of FLComponent for FL clients that has an -``execute`` method that produces a Shareable from an input Shareable. The ``execute`` method also takes a str for -task_name, FLContext, and abort_signal. +An :class:`Executor` is an FLComponent for FL clients used for executing tasks, +wherein the ``execute`` method receives and returns a Shareable object given a task name, +``FLContext``, and ``abort_signal``. .. literalinclude:: ../../../nvflare/apis/executor.py :language: python diff --git a/docs/programming_guide/fl_clients/model_learner.rst b/docs/programming_guide/execution_api_type/model_learner.rst similarity index 100% rename from docs/programming_guide/fl_clients/model_learner.rst rename to docs/programming_guide/execution_api_type/model_learner.rst diff --git a/docs/programming_guide/fl_clients.rst b/docs/programming_guide/fl_clients.rst deleted file mode 100644 index 754d787588..0000000000 --- a/docs/programming_guide/fl_clients.rst +++ /dev/null @@ -1,52 +0,0 @@ -.. _fl_clients: - -########## -FL Clients -########## - -FLARE Clients are workers in the FL system that perform tasks. -We provide different levels of abstraction for writing FL Client code to support use cases ranging from complete customizability to easy user adaption. -Here is a general overview of the key ideas and use cases of each FL Client type ordered from most FLARE-specific to least FLARE-specific: - -**Executor** - -An :ref:`executor` is an FLComponent for clients used for executing tasks, wherein the execute method receives and returns a Shareable object given a task name. -Executors are the most flexible for defining custom logic and tasks, as with a custom executor and controller, any form of computation can be performed. -However, Executors must deal directly with FLARE-specific communication concepts such as :class:`Shareable`, :class:`DXO`, and :class:`FLContext`. -As a result, many higher level APIs are built on top of Executors in order to abstract these concepts away for easier user adaption. - -Overall, writing an Executor is most useful when implementing tasks and logic that do not fit within the structure of higher level APIs or other predefined Executors. - -**Model Learner** - -The :ref:`model_learner` is designed to simplify writing learning logic by minimizing FLARE specific concepts. -The :class:`ModelLearner` defines familiar learning functions for training and validation, and uses the :class:`FLModel` object for transferring learning information. -The ModelLearner also contains serveral convenience capabilities, such as lifecycle and logging information. - -The Model Learner is best used when working with standard machine learning code that can fit well into the train and validate methods and can be easily adapated to the ModelLearner subclass and method structure. - -**Client API** - -The :ref:`client_api` provides the most straightforward way to write FL code, and can easily be used to convert centralized code with minimal code changes. -The client API uses the :class:`FLModel` object for data transfer, and supports common tasks such as train, validate, and submit_model. -Additionally, options for using decorators or PyTorch Lightning are also available. - -As of version 2.4.0, we recommend users start with the Client API, and to consider the other Client types for more specific cases as required. - -**3rd-Party System Integration** - -The :ref:`3rd_party_integration` pattern allows for a seamless integration between the FLARE system and a third-party external training system. -This is especially useful with pre-existing ML/DL training system infrastructure that cannot be easily adapted to the FLARE client. - -With the use of the :mod:`FlareAgent ` and :mod:`TaskExchanger `, users can easily enable any 3rd-party system to receive tasks and submit results back to the server. - - -For more details about each client type, refer to each page below. - -.. toctree:: - :maxdepth: 3 - - fl_clients/executor - fl_clients/model_learner - fl_clients/client_api - fl_clients/3rd_party_integration \ No newline at end of file diff --git a/docs/programming_guide/fl_clients/client_api.rst b/docs/programming_guide/fl_clients/client_api.rst deleted file mode 100644 index ebfe051c4e..0000000000 --- a/docs/programming_guide/fl_clients/client_api.rst +++ /dev/null @@ -1,349 +0,0 @@ -.. _client_api: - -########## -Client API -########## - -The FLARE Client API provides an easy way for users to convert their centralized, local -training code into federated learning code with the following benefits: - -* Only requires a few lines of code changes, without the need to restructure the code or implement a new class -* Reduces the number of new FLARE specific concepts exposed to users -* Easy adaptation from existing local training code using different frameworks (PyTorch, PyTorch Lightning, HuggingFace) - -Core concept -============ - -Federated learning's concept is for each participating site to get a good model (better than -locally trained model) without sharing the data. - -It is done by sharing model parameters or parameter differences (certain filters can be used to -ensure privacy-preserving and protects against gradient inversion attacks) to each other. - -The aggregators will take in all these model parameters submitted by each site and produce a -new global model. - -We hope that this new global model will be better than locally trained model since it -conceptually trained on more data. - -One of the popular federated learning workflow, "FedAvg" is like this: - -The general structure of Federated Learning algorithms involve the following steps: - -#. controller site initializes an initial model -#. For each round (global iteration): - - #. controller sends the global model to clients - #. each client starts with this global model and trains on their own data - #. each client sends back their trained model - #. controller aggregates all the models and produces a new global model - -On the client side, the training workflow is: - -#. receive model from controller -#. perform local training on received model, evaluate global model for model selection -#. send new model back to controller - -To be able to support different training frameworks, we define a standard data structure called "FLModel" -for the local training code to exchange information with the FLARE system. - -We explain its attributes below: - -.. literalinclude:: ../../../nvflare/app_common/abstract/fl_model.py - :language: python - :lines: 41-67 - :linenos: - :caption: fl_model.py - -Users only need to obtain the required information from this received FLModel, -run local training, and put the results in a new FLModel to send back to the controller. - -For a general use case, there are three essential methods for the Client API: - -* `init()`: Initializes NVFlare Client API environment. -* `receive()`: Receives model from NVFlare side. -* `send()`: Sends the model to NVFlare side. - -Users can use these APIs to change their centralized training code to federated learning, for example: - -.. code-block:: python - - import nvflare.client as flare - - flare.init() - input_model = flare.receive() - new_params = local_train(input_model.params) - output_model = flare.FLModel(params=new_params) - flare.send(output_model) - -See below for more in-depth information about all of the Client API functionalities. - -Client API Module -================= - -nvflare.client.init -------------------- - -- Description: initialize required environment variables for NVFlare ML2FL client API -- Arguments: - - - config (str or dict): the path to the config file or the config dictionary - - rank (str): local rank of the process. It is only useful when the training script has multiple worker processes. (for example multi GPU) - -- Returns: None - -Usage: - -``nvflare.client.init(config="./config.json")`` - -Config example: - -.. code-block:: json - - { - "exchange_path": "./", - "exchange_format": "pytorch" - "transfer_type" : "FULL" - } - -Exchange_path is the file path where the model will be exchanged. -Exchange_format is the format we expect of the model, pre-defined ones are "raw", "numpy", "pytorch" -Transfer_type is how to transfer the model, FULL means send it as it is, DIFF means calculate the difference between new model VS initial received model - -nvflare.client.receive ----------------------- -- Description: receive FLModel from NVFlare side -- Arguments: - - - Timeout (Optional[float]): timeout to receive an FLModel - -- Returns: FLModel - -Usage: - -``model = nvflare.client.receive()`` - -nvflare.client.send -------------------- - -- Description: send back the FLModel to NVFlare side -- Arguments: - - - fl_model (FLModel): FLModel to be sent - - clear_registry (bool): whether to clear the model registry after send - -- Returns: None - -Usage: - -``nvflare.client.send(model=FLModel(xxx))`` - - -nvflare.client.system_info --------------------------- - -- Description: gets system's metadata -- Arguments: None -- Returns: A dictionary of system's metadata - -Usage: - -``sys_info = nvflare.client.system_info()`` - -System's metadata includes: - -- identity -- Job_id - -nvflare.client.get_job_id -------------------------- - -- Description: gets the NVFlare job id -- Arguments: None -- Returns: JOB_ID (str) - -Usage: - -``job_id = nvflare.client.get_job_id()`` - -nvflare.client.get_identity ---------------------------- -- Description: gets the NVFlare site name that this process is running on -- Arguments: None -- Returns: identity (str) - -Usage: - -``identity = nvflare.client.get_identity()`` - -nvflare.client.clear --------------------- - -- Description: clears the model registry -- Arguments: None -- Returns: None - -Usage: - -``nvflare.client.clear()`` - -nvflare.client.get_config -------------------------- - -- Description: gets the model registry config -- Arguments: None -- Returns: identity (dict) - -Usage: - -``config = nvflare.client.get_config()`` - -nvflare.client.is_running -------------------------- - -- Description: check if FLARE job is still running in the case of launching once -- Arguments: None -- Returns: bool - -Usage: - -.. code-block:: python - - while nvflare.client.is_running(): - # receive model, perform task, send model, etc. - -nvflare.client.is_train ------------------------ - -- Description: check if current task is train -- Arguments: None -- Returns: bool - -Usage: - -.. code-block:: python - - if nvflare.client.is_train(): - # perform train task on received model - -nvflare.client.is_evaluate() ----------------------------- - -- Description: check if current task is evaluate -- Arguments: None -- Returns: bool - -Usage: - -.. code-block:: python - - if nvflare.client.is_evaluate(): - # perform evaluate task on received model - -nvflare.client.is_submit_model() --------------------------------- - -- Description: check if current task is submit_model -- Arguments: None -- Returns: bool - -Usage: - -.. code-block:: python - - if nvflare.client.is_submit_model(): - # perform submit_model task to obtain best local model - -Client Decorator Module -======================= -nvflare.client.train --------------------- - -Use cases: - -.. code-block:: python - - @nvflare.client.train - def my_train(input_model=None, device="cuda:0"): - ... - return new_model - -NVFlare will pass the FLModel received from the NVFlare server side to the first argument of the "decorated" method. -The return value needs to be an FLModel object, we will send it directly to the NVFlare server side. - - -nvflare.client.evaluate ------------------------ - -Use cases: - -.. code-block:: python - - @nvflare.client.evaluate - def my_eval(input_model, device="cuda:0"): - ... - return metrics - -NVFlare will pass the model received from the NVFlare server side to the first argument of the "decorated" method. -The return value needs to be a "float" metric. -The decorated "my_eval" method needs to be run BEFORE the training method, so the metrics will be sent along with the trained output model. - -Lightning Integration -===================== -nvflare.client.lightning.patch ------------------------------- - -- Description: patch the PyTorch Lightning Trainer object -- Arguments: trainer -- Returns: None - -Usage: - -.. code-block:: python - - trainer = Trainer(max_epochs=1) - flare.patch(trainer) - -Advanced Usage: - -Note that if users want to pass additional information to NVFlare server side VIA the lightning API, they will need to set the information inside the attributes called ``__fl_meta__`` in their LightningModule. For example: - -.. code-block:: python - - class LitNet(LightningModule): - def __init__(self): - super().__init__() - self.save_hyperparameters() - self.model = Net() - self.train_acc = Accuracy(task="multiclass", num_classes=NUM_CLASSES) - self.valid_acc = Accuracy(task="multiclass", num_classes=NUM_CLASSES) - self.__fl_meta__ = {"CUSTOM_VAR": "VALUE_OF_THE_VAR"} - -Configuration and Installation -============================== - -In the client_config.json, in order to launch the training script we use the :class:`SubprocessLauncher` component. -The defined ``script`` is invoked, and ``launch_once`` can be set to either launch once for the whole job, or launch a process for each task received from the server. - -A corresponding :class:`LauncherExecutor` is used as the executor to handle the tasks and peform the data exchange using the pipe. -For the Pipe component we provide implementations of :class:`FilePipe` and :class:`CellPipe`. - -.. literalinclude:: ../../../job_templates/sag_pt/config_fed_client.conf - :language: json - -For example configurations, take a look at the :github_nvflare_link:`job_templates ` directory for templates using the launcher and Client API. - -.. note:: - In that case that the user does not need to launch the process via the SubprocessLauncher and instead has their own external training system, this would involve using - the :ref:`3rd_party_integration`, which is based on the same underlying mechanisms. - Rather than a LauncherExecutor, the parent class :class:`TaskExchanger` would be used to handle the tasks and enable pipe data exchange. - Additionally, the :class:`FlareAgent` would be used to communicate with the Flare Client Job Cell to get the tasks and submit the result. - -Examples -======== - -For examples of using Client API with different frameworks, -please refer to :github_nvflare_link:`examples/hello-world/ml-to-fl `. - -For additional examples, also take a look at the :github_nvflare_link:`step-by-step series ` -that use a :github_nvflare_link:`Client API trainer `. diff --git a/docs/programming_guide/fl_model.rst b/docs/programming_guide/fl_model.rst new file mode 100644 index 0000000000..6b2a9bad07 --- /dev/null +++ b/docs/programming_guide/fl_model.rst @@ -0,0 +1,17 @@ +.. _fl_model: + +FLModel +======= + +We define a standard data structure :mod:`FLModel` +that captures the common attributes needed for exchanging learning results. + +This is particularly useful when NVFlare system needs to exchange learning +information with external training scripts/systems. + +The external training script/system only need to extract the required +information from received FLModel, run local training, and put the results +in a new FLModel to be sent back. + +For a detailed explanation of each attributes, please refer to the API doc: +:mod:`FLModel` diff --git a/docs/programming_guide/resources/te.py b/docs/programming_guide/resources/te.py deleted file mode 100644 index 3f511a6c34..0000000000 --- a/docs/programming_guide/resources/te.py +++ /dev/null @@ -1,9 +0,0 @@ -def _get_model_weights(self) -> Shareable: - # Get state dict and send as weights - new_weights = self.model.state_dict() - new_weights = {k: v.cpu().numpy() for k, v in new_weights.items()} - - outgoing_dxo = DXO( - data_kind=DataKind.WEIGHTS, data=new_weights, meta={MetaKey.NUM_STEPS_CURRENT_ROUND: self._n_iterations} - ) - return outgoing_dxo.to_shareable() diff --git a/docs/programming_guide/workflows_and_controllers.rst b/docs/programming_guide/workflows_and_controllers.rst index 052327365e..9a75c9901d 100644 --- a/docs/programming_guide/workflows_and_controllers.rst +++ b/docs/programming_guide/workflows_and_controllers.rst @@ -7,12 +7,13 @@ A workflow has one or more controllers, each implementing a specific coordinatio CrossSiteValidation controller implements a strategy to let every client site evaluate every other site's model. You can put together a workflow that uses any number of controllers. -Before version 2.4, all federating learning workflows (fed-average, cyclic controller, cross-site evaluation) were server controlled, -implemented with the server-side :ref:`controllers `. In these workflows, -FL clients get tasks assigned by the controller, execute the tasks, -and submit results back to the server. The first section covers the server-side -controller API for server-controlled workflows. The second section covers :ref:`client_controlled_workflows` for -workflows that are controlled by the clients. +We have implemented several server controlled federated learning workflows (fed-average, cyclic controller, cross-site evaluation) with the server-side :ref:`controllers `. +In these workflows, FL clients get tasks assigned by the controller, execute the tasks, and submit results back to the server. + +In certain cases, if the server cannot be trusted, it should not be involved in communication with sensitive information. +To address this concern, NVFlare introduces Client Controlled Workflows (CCWF) to facilitate peer-to-peer communication among clients. + +Please refer to the following sections for more details. .. toctree:: :maxdepth: 3 diff --git a/docs/release_notes/flare_230.rst b/docs/release_notes/flare_230.rst index a042e43212..6721de4b86 100644 --- a/docs/release_notes/flare_230.rst +++ b/docs/release_notes/flare_230.rst @@ -41,7 +41,7 @@ Prior to FLARE 2.3.0, model initialization was performed on the server-side. The model was either initialized from a model file or custom model initiation code. Pre-defining a model file required extra steps of pre-generating and saving the model file and then sending it over to the server. Running custom model initialization code on server could be a security risk. -FLARE 2.3.0 introuduces another way to initialize the model on the client side. The FL Server can select +FLARE 2.3.0 introduces another way to initialize the model on the client side. The FL Server can select the initial model based on a user-chosen strategy. Here is an example using client-side model initialization: https://github.com/NVIDIA/NVFlare/tree/main/examples/hello-world/hello-pt. You can read more about this feature in :ref:`initialize_global_weights_workflow`. @@ -67,7 +67,7 @@ Federated Private Set Intersection (PSI) In order to support vertical learning use cases such as secure user-id matching and feature over-lapping discovery, we have developed a multi-party private set intersection (PSI) operator that allows for the secure discovery of data intersections. Our approach leverages OpenMined's two-party -`Private Set Intersection Cardinality protocol `_, which is basedon ECDH and Bloom Filters, and we have +`Private Set Intersection Cardinality protocol `_, which is based on ECDH and Bloom Filters, and we have made this protocol available for multi-party use. More information on our approach and how to use the PSI operator can be found in the :github_nvflare_link:`PSI Example `. diff --git a/docs/release_notes/flare_240.rst b/docs/release_notes/flare_240.rst index 97d03179f6..d2fe661ad5 100644 --- a/docs/release_notes/flare_240.rst +++ b/docs/release_notes/flare_240.rst @@ -23,7 +23,7 @@ Here is a brief example of a common pattern when using the Client API for a clie # initialize NVFlare client API flare.init() - # run continously when launching once + # run continuously when launching once while flare.is_running(): # receive FLModel from NVFlare @@ -63,12 +63,12 @@ Furthermore, the Job CLI also offers users a convenient method for submitting jo ``nvflare job list_templates|create|submit|show_variables`` -Also explore the continously growing :github_nvflare_link:`Job Template directory ` we have created for commonly used configurations. +Also explore the continuously growing :github_nvflare_link:`Job Template directory ` we have created for commonly used configurations. For more in-depth information on Job Templates and the Job CLI, refer to the :ref:`job_cli` documentation and :github_nvflare_link:`tutorials `. ModelLearner ------------ -The ModelLearner is introduced for a simplifed user experience in cases requiring a Learner-pattern. +The ModelLearner is introduced for a simplified user experience in cases requiring a Learner-pattern. Users exclusively interact with the FLModel object, which includes weights, optimizer, metrics, and metadata, while FLARE-specific concepts remain hidden to users. The ModelLearner defines standard learning functions, such as ``train()``, ``validate()``, and ``submit_model()`` that can be subclassed for easy adaptation. @@ -89,7 +89,7 @@ Each example will build upon previous ones to showcase different features, workf - sag_model_learner: scatter and gather workflow illustrating how to write client code using the ModelLearner. - sag_executor: scatter and gather workflow demonstrating show to write client-side executors. - sag_mlflow: MLflow experiment tracking logs with the Client API in scatter & gather workflows. -- sag_he: homomorphic encyption using Client API and POC -he mode. +- sag_he: homomorphic encryption using Client API and POC -he mode. - cse: cross-site evaluation using the Client API. - cyclic: cyclic weight transfer workflow with server-side controller. - cyclic_ccwf: client-controlled cyclic weight transfer workflow with client-side controller. @@ -133,7 +133,7 @@ Client-side controlled workflow Three commonly used types of client-side controlled workflows are provided: - :ref:`ccwf_cyclic_learning`: the model is passed from client to client. -- :ref:`ccwf_swarm_learning`: randomly select clients as client-side controller and aggregrators, where then Scatter and Gather with FedAvg is performed. +- :ref:`ccwf_swarm_learning`: randomly select clients as client-side controller and aggregators, where then Scatter and Gather with FedAvg is performed. - :ref:`ccwf_cross_site_evaluation`: allow clients to evaluate other sites' models. See :github_nvflare_link:`swarm learning ` and :github_nvflare_link:`client-controlled cyclic ` for examples using these client-controlled workflows. @@ -202,13 +202,13 @@ FL HUB: Hierarchical Unification Bridge ======================================= The FL HUB is a new experimental feature designed to support multiple FLARE systems working together in a hierarchical manner. In Federated Computing, the number of edge devices is usually large with often just a single server, which can cause performance issues. -A solution to this problem is to use a hierachical FLARE system, where tiered FLARE systems connect together to form a tree-like structure. +A solution to this problem is to use a hierarchical FLARE system, where tiered FLARE systems connect together to form a tree-like structure. Each leaf of clients (edge devices) only connect to its server, where this server also serves as the client for the parent tier FLARE system. One potential use case is with global studies, where the client machine may be located across different regions. Rather than requiring every region's client machines connect to only a single FL server in that region, the FL HUB could enable a more performant tiered multi-server setup. -Learn more about the FL Hub in the :ref:`hierarchy_unification_bridge` documenation and the :github_nvflare_link:`code `. +Learn more about the FL Hub in the :ref:`hierarchy_unification_bridge` documentation and the :github_nvflare_link:`code `. Misc. Features ============== @@ -236,7 +236,7 @@ Misc. Features - We added the application layer ping between Client Job process and Server parent process to replace the gRPC timeout. Previously, we noticed if the gRPC timeout is set too long, the cloud provider (eg. Azure Cloud) will kill the connection after 4 minutes. - If the timeout setup is too short (such as 2 mins), the underlying gRPC will report too many pings. + If the timeout setup is too short (such as 2 minutes), the underlying gRPC will report too many pings. The application level ping will avoid both issues to make sure the server/client is aware of the status of the processes. - FLARE provides two drivers for gRPC based communication- asyncio (AIO) and regular (non-AIO) versions of gRPC library. One notable benefit of the AIO gRPC is its ability to handle many more concurrent connections on the server side. @@ -278,7 +278,7 @@ For this financial application, we use the `Elliptic++ `_. -Finanical Application Examples +Financial Application Examples ------------------------------ To demonstrate how to perform Fraud Detection in financial applications, we introduced an :github_nvflare_link:`example ` illustrating how to use XGBoost in various ways to train a model in a federated manner with a `finance dataset `_. @@ -330,7 +330,7 @@ Here is the default meta.json which can be edited accordingly: FLARE API Parity ================ -In FLARE 2.3.0, an intial version of the FLARE API was implemented as a redesigend FLAdminAPI, however we only included a subset of the functions. +In FLARE 2.3.0, an initial version of the FLARE API was implemented as a redesigned FLAdminAPI, however we only included a subset of the functions. In FLARE 2.4.0, the FLARE API has been enhanced to include the remaining functions of the FLAdminAPI, so that the FLAdminAPI can sunset. See the :ref:`migrating_to_flare_api` for more details on the added functions. diff --git a/docs/programming_guide/resources/3rd_party_integration_diagram.png b/docs/resources/3rd_party_integration_diagram.png similarity index 100% rename from docs/programming_guide/resources/3rd_party_integration_diagram.png rename to docs/resources/3rd_party_integration_diagram.png diff --git a/docs/programming_guide/resources/fed_sag_round.png b/docs/resources/fed_sag_round.png similarity index 100% rename from docs/programming_guide/resources/fed_sag_round.png rename to docs/resources/fed_sag_round.png diff --git a/docs/programming_guide/resources/init_weights_1_config_fed_server.json b/docs/resources/init_weights_1_config_fed_server.json similarity index 100% rename from docs/programming_guide/resources/init_weights_1_config_fed_server.json rename to docs/resources/init_weights_1_config_fed_server.json diff --git a/docs/resources/task_execution_decision_chart.png b/docs/resources/task_execution_decision_chart.png new file mode 100644 index 0000000000000000000000000000000000000000..c40bae817e653bc38784e7c2fa742322b70b4fde GIT binary patch literal 73494 zcmeEv2_RM5{=XpV$UG(USf&7aWnE><>mrD__;ZR1%+<_mx26Na3|-ZykL8K z4s&a;r6btRi30|4KDrLJgusv)l;9rLP$-y@M~0t+3%He832bH!bA~%vG4fsm&gEdv z5Ge2sBm@;o>;_2PA5MEx?Y}$BIDI1mX;~JgUhxWlIpR zin+WdueGhbpp>z^tDMSFZCoLaPS$W3vT=AgxH)*ZkFGf1w1*reTf*U%P~eG(GVOut zJ0Wu;Qh22qxvm1~h+keQWNQH@*2wmDx&~HJQROpL6Lip)lLgswvzr+ol>`F=wK(h( zbBLMs;Y;vvnOT7y58L^8xv0|Z++`iX_Et)8^TSp(cSpVlmmuF!k>)p#Qn-&a8HK!vS4wQGS|Bfyk|3nNUDF_sSVN^RJp94VcpTGB)YC8Th zg#IVRNhv$P!BNQ8W3(WkGG5}Ox`!$au9BvP^EkGRCY#`3gH<3sJyEwyvBr9h-DDo;$ zKgXN8zzG)zKXB#nR1a~AI=PMj@2_VF;Qn%^3URcygE&JRk%>nYum}IzYRHCkg1b1H zA#3@=#z5wC20K~;^mPcTALm1M1EOV)vK`*}_rZa}5il$v03!hv24La{fr6c_U5}T- z!v#hT56fTj{gyb6L+<-y1PE2(0IveYBZs^Y@_kX`9dQ$4xT79)RAJ;zhzoxV*Q4k1 z0*otjtU`ak1(3N9N5~)L{B9uqxFfzpF9-}YCQvir4+;ku1Ne4!1Y5(bVV1z5lbNHn z{UKoj$Hx#3g#y$Cbiy?^D{E(nhCTQYCT;+I{uZYeaG0|U*v=Y?Xh{$R>I!kTHUk5B z5Eqc><_0?J8W?JAiI5Uz0F)8Hyavcws1zIuM}U-(hu6XaB4B0)q&qn~!fhePu9yo7 zn*wx&5QvDL_yLRo8RQH>_UylkJLK)WzXOH<^4#rLR{e!Lt&TA&3Q#Bj9Y^Xvf}TGC z3IE%O4CujMY3x^4`x`{|H#_@}ISbik$4N_2P}*2e+X61=q^o*eScuozLdBIGNp3$2 zXK@Sv7Mr~PF{H(d?1^JCR}avp4u|_OQUb;}AcKxeNk@p2wFlVrP%a}F!XCl$hq8kI z8YBO8AmuySL2w1YP|&MLKC-hmH%Hum4GK1eK(B$#Y%LFk_b~#pIQ#*!e2>W=b1-V; z{hpQp_4G*pL%uRl-eH&iedR%Fh3rXR{+; z1x12z3kxTd)$`K`P~oF09mC>L!Z9HJAJYGRz@Zs*;{c~K;dqOz8{WXoIyXfUH)N%1(ETX62Nu%1H8m{8pU@8i?DzoFIWJ` z`a2T`!fnnE5&UKF{ket=ijYxi0gCXB8#1Uo#|;_4DR|TqC=mYtw`6!w`qY12GJt;y z)jxk|-UuG`<4-D)1yQb56b^r0iHvkEA7ey>L`G=o$8^e{MgIOW82^0v3m7+CJioIr zke1?~ERc`;l6bgK;+Y58ktjs|jN+LGIj?aHjNd`=FEwufarl>bh5ljMMC#8!3vD7@ z2*03|A2sKLLg~-To8OE5pV=$)%gFoli4)*WVJ@Vdge0%OlsDPAf9sQCN4^URl0PMJ zes9-Yha;SnTvAYLfWd%+tEzIqryJbS_P^%Lp?dRgoR>k-HxkZ#KeIW5RLqZwU}<-V z888J5cxQg66#`S||K@qPf8DG#72@Xugv)nkt%Z;oBL z*P_yn+qEdbpg8C+HEaLg^wTjI@H(0U1Oham^G)Czn96_v&JAZobj^QpfgpgxU&L&_ z>@RTus0n?Z-{FE6g@K)CNcz{* ze`hSxKWD5#+^wB;f#?oo2orFo2b>{dpAc7mo%ctaAm|(+TM$?NrSPR6QY=!^9(R@f zc=qb&_LT{v$PXpleg>QGhaL8_y2^f78Pw?g^9{cLddmJs@l!wSJ*4CDm>B+W|Mepv4xaOMy5Ax-?B#!rUgNTlH)D2O!Ze|paR-lO}IxXFG9E7Y+2^GWl+Ub25b zX@2k8Ksw0~%Tu^44nKiVcVq}M5DNg%tF<}U84;lj#KxIHfLP@Jh5{fgyg$!(CV+B< z95;0RAmRNsY6|aob6;3)O0M4kn`(`G8t(Y42C>&!6QJI(p_G)*jhP$Bm93Rtnil#~n34%!e%RFElz3L;H_cDgt?s z>$AS6gg&x4@m##VD)d*k$$)C3w3&2MM8*ok5{JW6-^G$xhPl|NPBE3K;Y5Ld16Cy=7>e)zC zqqyULc~r~~yY72??EetkAP3=pJlmjF-257T|C2Zv|1M3}=Y`6w8{fy=AgvT6y0}rx*N!XAi2DK1aUEBhfAI~VVEF%EYvx9^ z&aWKzA0Wt(0_qRdX4KT!f4n3^#U&pjkDpnT{es%xkLf>UqX9laq{;OE$w;D5Q zx#Q1FI^R2sq~I_MV0$}YzWu+Nu@Gt>sDGn3`S*+RerDAerNJN5nf?P*V_;9ZL%ZaU zbz`In{hu$hkS%hopQYgTh=nM?o-trMd%!Y=nK~i9wA2wBcz_RQ$gN>cfK&iZ4?}x^ zB`b*F9(%+NxI8k4Cim|{GJf3|6Jcxr!R~^8IB^8*%*FFJb`2CT72roqJ{|S*KRQ$S zCz1d!YO?5$?Bn0>Mf7Ky$67%5shHXv1vwmUfp)kZBM%@UfGt~q9Z-Pxv4;^` zFek*`oQLidXRy;@6a;b_@^EKRAdU;z`SI#ex61(gz4A!??cIA(+rj?OM|b46`bRwz z$PH$XZKm*-%z{cw1L*o=|IIJU=KdJWe|sU_zulwrcl|o3*-xYb#)k@(Jr3MIV>T4^ zQvdjDI*{`ig8x2X{?WO2Zlsxe98Z6KK*hhZSmvLjwm%HQK)HebSSJ4UcHH0Z+9Tkg z;B(c{5sZ*qD4iX-E94JjwvONV{V@VEzuP(Zm(^s*!NG%KbpVP#hza|?%;PCPDF2u8 z{%7SMM#LR8#6Qw>sPHyhr=R|9(q5RbV$zCm;`?t~j_Lb|$~3@arLvepnFnc&Ugzkx?Zhl^VoDOySPX za67~uKwRL_#KHevErt)#pul)X?SjLN(oT*G160~^FFz^`H46VsE#|wuv40FQzN4BP z1KCl+U(59WX4XbZtN)TVfYZa@e}GhZQ5w!aN7_FbsUnea4DQIV=ELwoh=YqYaOr4@ z^vGg6qR1ot{b)M%DA@V%7{LL@2Fm{-JMOmx_=|u(PJk$q{HqD@5Rw0`ka?t7NBjpk zN1BWP%dLlKXtZc@*RN{28!aT_=xPk?G}Xurda~LWW+i92Bx}!)=W45|XUhNsWUBOdVvD^k<(l-}(g z#Jv_le&U)LTDu*&xjJ`Rq$21|cKH3M@Fllket%Ap)m!u_I$o`~M#U-CRw?rTWlBPWSKp}fHEL3NW4sye10 zdS8!^_G4xUXz^t@X|K?mGu)+TpS3ZysH?#U6vDrFZvNY3@Jsy@*Ql~Y$XF=mA5&by zIt@Y}$7C_(_ju4Qzf0SBYoNET5cbUFvnzqNa>$*=^N=9BhYg=2Ko80pPY~G0B$$4GTgz}d{~ek0 zl~D%qQK8YFqCg%bL4!>z`{Yx)S$A;#<)>Bf76M`3((mbQu@v6Sy2?R(oyg?PV;fXr<)*OtpPU zC4+QBGOoQN`bHQ2N>rP|3TB-0QjU9GeR}C!(sFaKC(%Q!N6gpa*Ib|Ns6J+MOzlNu zN&bjS%octjPnI!7E!&47GFV1gD}ZuD=ZnBQ!K$ez%k9CQFXS&1=@~@MR&G@XNhEzr z2%9EkN#?)`!Uu&3#w8_0k@NT0_DQ@cpCc5QbG%=$*l=Z&iDln9IV8e58!Kj~=G6P# zL~~LW)yKG2)SwT!Z@UNE9a6I)j1`gmOd|6=6l!WXL3!NL6N!5qqHA)ILcE<_X3QL< zB}zFexSyghZeaCHu|q6^cqg#R+)scOTipcvnCN1@GJb`C)WcTF||hsP!51qS**V`h2IOKw)>Hpk+M8?!u~$`?uvZGC9}Ds)iR?oz7RJ zHRLraUZwdq`mI`#XwnEsc5Zytx-(hI_F<^B*Nre1m#$+@-Gy11kV02(Hx$ zP+0%adNfGmrH`6VL~~Dh3PnfXaz>9S$JAPh8c|Aw99-}q*ko@#wvrx8Tw8Kc^3_Z1 zyt`+pI7dtZU-tx>q(QsW-xxLK=1R1rg_q=spJ0l~Q<;%NH`JEQlf*Ulr?{5#Ifkdi z;^B=L8p+6Vbre%x-H zHD_dZ{N^d@KTU_VYUp|s4t^DLUl-Mdrv5>cy-YL8%E zn+mm!p5IeUYo`{r^%+ksr)PYxZ?u(CD!MtS^k-hGJc(N{xwFpZ?+{PL55fvMk#--y zR4Q1BYX8~bmHVN0tb}eQC4LQ!GG_|T<)L**45)ZI3k)ueWGl%VaB8iLPAKW{Zr0XU zvxH0ZaK&KP4HcI3t??`z^<;NqRdQfNh=lSM<#3hWs*hRKR#GtAm|M9|<}myKD`vZ? z{jGXjFFs4M7M3PGzl@=_U0v5SMi0|6;#HoucF8Ju3AMo=?2=QUa^yR zQyp$-{^Yj#6vi})KNdK*lwZCU_iV(Egwo$(c~v=mMAPl=W2#fZZC-bAKHmOdlwCO- zuOnO00uQB~-LHGzx@R_@+rec6P*Na^VzM~`g%Ey&7M>{{cG$j%- zjMW6phL#!rtm|yJX+sJag~FR$j*gDqm?qxk_u?B^tGb^_w%O)P zHBx>0(W4OC0vFdCDPWjum8M{5>Wr^jJ9J;1ZxdX&9pz75P1$`;f?(mznqK%G z&!gaknSaYJ}y1s(~sd!7D+=hCQ+pT5ekaz>F3Ngh-fn=RY`YKzVzbpyZcz#Nk+w$Uo@kT8=$kXw{`B{t2 zsbzj4?{(bH1*5wKB80w)A^eSd5+=$?6+sel`LcEXs!Q;joEw+M-cHVYkPR15mM)70 z)Ms)`!afn4JaHfwK?+jOChpT;#V8)gv=!Ygw8tlkQn!b7QNUzKzcgKg?pU5Dii~`F zkZJAiag#>zA#Q$l3S5tbZ_6?6+*)r&5kc{&LO++!a;kI?}bE(cI_70 z-fS&10!h!Xq*>a?uL~F36%TfFVMerer$z2}KLD4#kO(hctDX9o)&Gpi$UxLLNvi2X zf`AgY-Ky%6U@7+O2ko<+q3c+~5;uEIO3>t3TN>(9l^{I>JjHYGB0Am;UMvQor<17S zf;Mzg%Cke}mx4N6b5_e5pJ_}-EcPY5e7O6O2RiI>vR`|d_fatU(gWkB)Ag^7y{F#S zN>2BeiB8t3IPg9-s%N@2y}Kg8WO~OqjBtOyYnyg=eZZ#jLC8JZEgtvniz`u*+b;X^ z(GCusUW)C_?YHmRn%$uux)UpQ&usCw^_QiU58&Op1s&4xhf-ag9!-{v67xoOWSz`n zbDDm{@3S%F(LdMTc!xH^L!J5Rmaz1UlWh>%=Q!hox*4Bklbsbk<^yRK0)f6U17TX~ zQ&soTQ!~ukPZ}Fkd*f6Lbk-O0#=WiX3cFwJa+%cR1K|@#mm#RMccGr{OhE;>{^>;b z{79+$<|>v=$q2BEe;8lYzN%1KtIPm9~EI=3Cxp?K}@&cf6Lt|6IRUy3@@P z%{FzLy&{a*sz$_**vnGJ(RA8RA_u?qQP;6n1=tNo^cDU*Os7-fbLcb)F?x@nw?wMx zP>(K;6V>2F=LZ++%lIEpV9H2d?({QT(+0^YeWb1RJwy5Ao#24ur=2}>82V!|@%V&C;Ddmgx5>&bON9kyH4#*NLvU!uYE#*A z8M*$|bz_NFr;-+Tav?q%9tLC=4-(x+>?E(T$Kd$0Mq*si0Nz^q*-H=ol&={?mtx}& zR-oFh0^`@4o?aVLAH{H1G{B%*fYK42!i(Wz7sKo_s=oS2y%XRbmKRj|Qx4CI{nGCW z^As0FJc?O+J5shAGf^4x!2%M~>Vr<7O8D+>v7YfMCeMdtlpnP<-grYy=|vYwlAbId zbb6WC$U)}R^wfR3aG7z^TpB;e+@RqlCC{zA(_qQ&+$s<#gr%Zf+GbC~kg+=>!tZFl zkVy=TIznwc23R5o9u~%SGRWaxjN+x*dMW&Lt$VFjTtS{(Sw1B!$*03lZ0JbcC~0*n|I(?mIs z^nO`ud^xY}+)<)Dy6nr;TGNFXt>`S>b9Q$xqE2o1XO zitKhP39e7EpjR#H6`wWic~WJaGCL)bD-KJn+k0GCa5&A5Icd(Ph?iGA+A^}wrnx@Q zKu-q*YUU+d` zz3(dc9+dtA)4a3PQprL(jgBhSa!@7+tSG0|?p|PR2p^qN) z*N(OmkBOGl#ZbBxn!RZvzcPDy#MfnH{dPyUu6~p({^a_#LNkViU_(;&H}Ge7sb1M~=f-ndR017(Gem%1uAzYI^S1JCwJU zmqTI_T+Lb^`*c@J({s?AZDZ6_eRXzS)YIIoiCfnuvb3t^o=#J&0p+`o1$R`L8-~LB zOmIw|R0TH#2}hN!K;8I7U5lJ190;@*_#_lg)rH2jC-M*}hq^xGa$@1(;f>h+vVe2m zcx!{BdW|Ne$`9|$t_yJhR{QHf>b7$78y(tJh3+zFIhvOrsA+sOo8y!!`{vkD8Q|zu z1U&&X(r(0J#i`=*(`X8oq5hsTxMbq6OELuOSMS9d`y}IhGah2wUJp)76O^^oqK^K$ zA5LwU=6iuTnK-=c-F|;Q9ZACS?s7xU?Tgc1*TKcnZ){)E&&{4CV11$>7)c6Z3G1wD z?YS*#-bOISlRS;nlbuXrHB@%tW$|!h9(mlE#o4NRqvgezl%$U5i-+3aK6%?GL05NF z>=%3M^j1L0)hgNk38`xyo?8-XQ&KO>si8hGl<+*vH`?v2f$FOZZrHG|@r$F1!H`JvfpRv%?(&1I zFkglU5Vw2#^5FAtXl~+mCd+Vc&z>-Y;V7ce_0e{=<(QhpT}e~kT&$Y?IxXW)Lt4&#Q0s`3U{6yzeum83b0<`I*pylFq zJt08}EQZ*ULeugd17YFtL9>f(xX+zto{UZmd2}bgt?@dMMp}?+Vq=P-p<(t|$dg1u zIq3oURE<>ci0;yv#iZ9*279)G_2ttJryXzB#?pW;i`K?tX_SlDmM+y!OB{R{e@y}j zXyB2N8GGbaAHpM(b7n>JE5Q zCpb3c&Hn3b%V&iuA*VJ33KF{rH?39SFv|K&-xrV#@ThPJW@DqwCw?eK95HNaaw0;9 z5Q53D(m1^tJeZ4@a~kw89csv`f^d$c`=!n4$X@C&YOIno<0Zzvy%UyX_AYWl*bY}= zz+uaTmcsQyb6(dKACkK&=z$$j-PRkWS&2RY&hyg(?wSrWkvp9FG!5zaV7ytuC7D|< zj6c~aUYT?GGQ*=^Br8zrW&=oJu%+4ETC@Gv+tfS433}cOiMjjN2lMPR zj=pyN-Di=(1o8F3H<8Z@bv6@RsT(@?n#vn&w>T~E6ztm6A5QO`L1!b43vU>DaxjIC zg@vZyYcZqSX%{Pg7ZTO(=_MalhY{ozw>Y*ZJ&?N^rY6P`lP|+Il(&JK1=xA~7ilyo zUoyU0ow>1I+E()6<>)mY{SC0^-C30#@;y)2o!wQ=s=mYUPh z4dNYI^pV@ip=@IN*ca<%YFxglY}x0ehc;#dPBa|cEP^!rb##mV7>rf{gAeVP5~u6# z#Kp-rSfq<5e4HhfO`yX!ZNTr6i}7nn3(`f}oeGv6oqVU^y5j3X?6W)xiI zj%0-IDs8t%c;KUVqd^`1(Zd+Y1*SGV1(ueTQ;tRBSY=z8RR^Yw2!3K!WuWC2yI{w0 z^G$W7n^u$jzI-D5ofJF&tomrw>*$7$6&Ey@4z#y4$|#D101sJjf#h4i7X%lGvfsn* zW{hBE^vCqVQ~X{&j9EF+)wR`p{hD3~tN$6;d&lq{k9^4pzlPPDmFex8Db1$lZ!AV0 zuNrXXRctIkpXg=_K^$iG-}+cpIn9L=cNbyUpc55uwaoT3u|KBhb=`8ry(*{VPv+t& zV<0oyM;WtSWu2RGDVrSOcFe*Dbzy*G>0gA|XYmZaO}QP$SWwlEJ~jEoPASBPiVV)6 zWM3e3=8=dA&m#~nu^xH}dY~RrH8%g9Ywy?~HeFmV9j&wbOCKy`6Yz%lxoXU290qlc>mu6_n;XoZ#9fxq9=kHqGs* z*8a?uG+z$3WLH{l?hDxQZgKKOZr*^4=DvUu>xB!@Z&UNOsSFr?%_Cm+uJD zX%Oxls_6cNjqqu6?3k~Q>5Dmrg1ii4@>Q6_iCpA~Zkko9(!N;Ga#5Y*AbLQ|bsdWj zo8i)|f1edr%4hG7HWIB=tg9(^m#$sByg#(Q`VdC}&00GyN6Mk~3}(cwOIQyvD6oj; z-@j$myq7&jm#fasZeXLB!H&K)$+6w}>|yUG4f1CB1x*OqlA3NLM!++`-PO>54|s`1 z72+^2YU}`BvUb2r#=Oh+B7lc%=ySoDt>rYJzdkFQ&LzO-@MT}6Y$|(4V%%j(CZTN+ z^(iV@xa?rD`Y`eWI#Gq=SO>m~ZlqCwxy-#Tk|0I-%UP3@!Y!&Uv^8>mySP>)pl9E{ z^jTR!C(GXUW<(mcP@GFtx{PmG8lAX~3tw&zZIQnj$=n(G?HgH88*yYB8q1d_7&mYt zE9aZ^<-6Ki?o&*?7vUNzpW7(i7H8%rjAd+;FofHhYi}2{R!5kvjTx_m5vxl3nPL*X zd|!1T=0sz9teg{1wW}*7%v5K5M3aPR(}3yg7SIr~xy4u)ixrv!w-K()FGEH}A;iqe zSV1Hp$0mC1)a>rc>08YVO9V04@gka^?6e<0%}QOq^KpcCGU-`YfFvj5s~B;Wnu zJkI;P6R`6q9g^ov(yX32**qlXqQ!E?25mJ>FX8qQky8rt@|4;!MOpMk%%+u-CB!Ln zm*$-0wbkJ@d%)Qq-2NDBK6f&rko``h_L-n|+A|af%MHP&d9umWsgP!MFLS)>jh`{d81YU#b>lf{Bo+Hr zGBV2BINbkUHpZQIg#>wT=v~a;<9@;qk0zHzr;W1yU{c5XK)Hn}f2hLlDPoxM#v&T;z4Rh8vC%1l+8Cosm@v zIKH9g>zhXMDEl1nCaTwgn=i7e0o+O49fWa$A+K36F&rb!_abmJv07j&;#eDa3u|eM zxjV$PSj32%@5^fj0LL#7Z^6c+!Gx&Aec)zk&8Iwwu!b$&I+tGzBhrX^oTk_of$waiuG!*&-mTx zG~NiOt|WtoS5qomMPq4B?_$eiQ&cN`%V~_@4SMBL|&6vBUfnIeY zvWV*<^Jh!@D4kcCzT@a}T4ZOVojKiGO!qV-3w$*-fzwsa*Dds8>XLBZG!Dd7Hs6*T zd+L&$zG*PTRkr39$}u&`kA7(^h^tKZTh#Z|Bs2ZetPxk4x?9eTsY+VmoQ(k~uiy z>biVn%$_9cS&48EgfM-m=nKcrM&>uIlSY_<%Rpm&<$vbVl}~SERB|Gtu_L5Iakzy+ zxWuWb$WYM^s&P5-v3abrvc>E=!R)2}ey+k7cM8jeuV-YOA*YmLCfb90Gx%#>^t*mJ zfPoEfGWfK4W;&fHBNq9ouaB%93O|45F05;xX0VOOmV41WIBlUUv#lnVF%7*J9Em4+)~7h+1Dkn#SQZB?8>%z0E9a-*Gb07Y8=?YML15KW@l3mG+!hPdCwen$}1q z@3nEBMNJLlzJ7)KbDzRhWs6I}?H!$#ml)wK=M@N@$6j(unNZ228H2vgUv3 ztMU%>a9gx9g?y9JmItQD9B0fF+}(wQ%Cw2%hlWTJcpn7@-uKzQPIT@y37Q6cr_X%#gt#Q39#XFOkxmM)L5@VF(YRB|3r;nD0 z`$OWn$OUM_m9ns`1VJ;6r>U0}FSEO4Ffnz`6Tc8Npln>h1-&8wdR5?>w%fMQ>|)<& z!|=A7gSLfO+zpooV|S0?_z}@h!I(hv-LJe5OXl(|BReA7`k@fNi%msxIB&eOub^ZP zDbQqm6a zp=TG%(4A{vb(Wb@QAS3UCT6C#70ENh_~_EMi#5YKTkwdpSs6oi{gh?DV4wCZN?)1D0$r;2U6 z2dCv=J&SQt+2>`liUI^Xq}9gQ&rE#WsEEqpxolVxhUG@iu%Kj*W94@_WxdE>=o!_L|3TiNIHYqZp;f;x>w2`^IC>vm? z#?fUKw?6%g&Yu{)Grp%ejHJr z1$iGpj`jIzKRHn?NGwHGRXr>;=S5v)cA)v~inRa&QeBaE(;yOj2`wh}OW5FComK`6 zy^!0H)!F!<6ZC5Y)K}%IJXnW^KVS9Mk@xh_W*f)?99Aw_j~#IkGRh~@lkB`00 z6SU(@7XVPiCk9;}84!p*eHN67vr1#R^ogr97VdLHs7G0Ms=r@}`|~LWyc_vrS7gW2 zTMs~=43e{T}DrOpJ$;x6btEez*jbLI=-aN06acO;7m?`|M}>A!-`9}#x>#uR6@R; ziK0%k+gn>7T4UJCoIVmLMsMB<<3Rg-Kg`Y8)-qE%2@~hr3cpY2bqxCTyEnut^KtS+ zlstD}<`s#xqk`|FljvA5Qt{%tQkLHvVYRyPuVTx`He*fVK=zFUXbR|!!L`q%@5=kE zHh?u<3IbLjY?2WxL7^X}GQ+xGDe(5@YS7dsYE88F+CFtxk&%_KQqz>Z=gAu*8}0|Vpz)KvWt4H41pHxEds zw%6uA_2*H>av0+3>+7eRDT29sHNQmHz;$Q(?XDDKCbz^Wvcp{*1ICZwYmJ>6kmBo zUM^!ux3c97rZN$}ogJAq_{x?Ed!0V*Y1PQokg6OtH+s;Cm*ynPnL0{3BVi-X$Vkpl z@oK8>Yp&I3DSBxh-fhpA9QUm`EZ4=czz$IX)BX0lUzYkQYd!tBoc5PDH%6(YJEmK$ zJu(`6qaQyGX4T3u8GH^Ntnw6$=eG@*X^S(fyK{;vc((nK|LUsqX)-Rfl$4Y#wN&iK ziHU_yAC(73M$FnDa*x8z8iPOZhZF2L=bA8ZRZsAlR5n(hx!Boq^>b`q_fv1LP3MLC ziL#*=1-Q>L4dm}Hl-W{%);u?A@6h?bO|?r&$ds@(Q^R|lAjrfkdnv!)VJGP$F3y9e zT&;L%q#IAgi}*!FDW$xrd>io-OgEF4zpfkdRnv%_uG!@}9|1R-GK8M?rVz)4UU~uh zMlpJ-CS*NN^nF-M@$-vcXfEp&o=KOForKxk+yd{@zh4}{-}6{&n!{&{h>5vN6R>~c zNz&f9|8xW0Wxet<78w?wx=VRss`6N!6Eixpy|num{k`6bg2B4 zAk0wbBQ^7ODl@q1uihO{~a_6b9p73tNzp zkAGL*iot!D?atUazfAnQVcAAUtIzxz_!h2QFvo zQ!dNJ8@&Z;m?H$#jV070&vzQ++Vx3KMu3{HLie}vX*$BM6&SqaUt%eri_? zMm3wSe=j|XDb-cWEq{U2Y4)l5P|;=~oJK26t4w(_Dg4f+rOL;Wp%-?g*J_`h4qm~q z{#xlH&XtqqB**vcfdAS$&eX?K5|h1tPMfx$uUxMQsq4h2(|xhaSs8RIFnh{-%7XD= zw*d6lt74yoQ->@OA(Ob;j9&2-VsYP8)YHwpvJsf<~pR~Wf11~EtXCXGjl`V+JOQL$e zQ{dU8?DBYYw0CsHr=!r2m?)cAZ)m7&>pi+fo{@j_3+KmUXYS((!jjkCr(~&DQ_1XX zrMT1=yzei^^ZqpD|EezsTvsQ3KQQdzB|ZU)knm%2{YTQ1bJoMM?I(38t>=U9hT%=;-_H z?TYz|Yv;Lnn*c1MA%8~H)sZ)5*B0bR%V=1AQsdmYO7ETXF`}+AyKlaY zK4aq+##*!Omz*AX`AHOZ1&3zcs?qm_$9BP|13S|Ey~;4?vjVNF74~KW$_sd6A+_%^ z=;oZW8cBN@O}RU}66h6($P&DGbT$3djA%(#n=jFXm3pRa~_!%Jw_#|x=-b5fv0^{p<9ZAYXW^&^uFU0lR8R6l0;wVyy-QC&g zo7Z`NkO)taeB~*OQ7?l(+n;rr{c<5#pSHAM1k0@}=m}O&dM+uK&sQp#+7$tg3`yF; zp@BQbm_oE9GZE~|W2rD^5^RQk%T$>V)qDftX9new3EU;{i3=K;%FuC!_uCz|qZ10{(Y2H1 zZQ_706ka_oX*w?f1;>l+7tVJVVv>o@#HnCdFQq`DV)=s#2{!@a7UUp=P6Cf@Q228cX~2?op;vlmK+w=<_8`pK|=z?^2sSF{l#7t zVl5Lf%G=M&`}z4pZ=Uq7+&=?zCI0iS(qFNPnV!-s2Egm2wvBmKH7Y~!*VxU;hX z%xs9-&va5l_npqS#<1S?o?=Rl&$ZW^%fO5?TPsG^{IaQ>;> z*sFMpvjh9}QaDkmu!*hI*Cs5u0M7?j0Wlw1mp=wgrO1Ar8az9iI=pGz^m??Nic%Xk zV>DGPTUxt!FD1^$4Hi&8Zq7|x_{MI1uy3e>dfQ{;BW=V%L>?{eSXfyY3-2YTZ zqR!EDoa!$s#eshqFQPszg1s)(W&zT$RKE6Qwu+{?p_K3WB37%>-0hx^t~MX1r{8~_ zY8ssyA>A?}Eq()EpS)d_Vn+I8gD!DpnZ_W+8n8U)t=2O<&yxq*7`}3poExnqzoz{7 zy~VYNOy&2r#g;T^f$Y9+Wa1bV``dW@Ihp|zX)QuGx8zg8pq_D;Wi+~8JuGrRbvu;0 zsW|oC2QiP+tkE1D9WdAxcVAbjdml2Zy|}FhRhJ^zIm0Q|*516oFvguG>jtYFq#tP> z@~IoDMehou_5(3%3Y^wrJ(z8*ara(oUkXEb%(!jEO9-t8GQ`NZ>PYFay}}~sz5Q+- zbk#}3O%6TC98KUYB@FCfAh<9_D2u~1I6P<%@4I3<&@_6txDdK#3Tr&~^%J^|cjF6* za_Pk|Ut-5d|Jow02Zdl;+Q~_?x0Mcfqg6f|sXB{}qJyTzF){obLM`WO$gd1;YweS( zd-vh2(cFIbQJHS_J=PLazs^_&O@2zDY!|v0_EYJ1+6LsX%sy!m`9UM@`C(wb4n zi5ACUfGn`e(3YUR-cK-kSjB|1TBTK`zrnfKTJwb6YGvEIQSsRbb)Ab^p(1(kh4=ST z#5~_mb|ie-e%CxIE3p3QjRhYa(}a&$mcoPCL7#0!Kgn%yed9zgGtoW*{bKE{)1O9Am=F7=EFuR{GBEm3uQ&uWkPEstG2^}0ntF+GD|xYjw9A3hd}EQ&H;sTS`yD1WWhrN19`#HWQ4|r_4r2JpJEIcvMBMYX zTOnMn0EW5VLbotp^S~4DCHT#Gm-yD)(jZ($F?1rzC`&XlyFt@wSrd`*u8($Ae*W?g zMXfAP1^Uqovxn$c?vJibc2GWi6#q2MtTqs5@aZ~GQ3s29kLN{D8HRS=1TcK{si~=2 zwKEGWkyU^WeH}U?gU*|@nPxtCL*Jq14HILEb6bB|g{2^C+`Dn^_~1?VJ6QQQIJIj&V8oOb#}&T7PGC+iRk4stM&j<0_5seN$9(1icOvy zig3O;ODB5Xkj21W-Gyd9XYY41C2NHrds$8dEcgW$IxJDd4pST`N(n!DR90pM#>Qdr zx*K8OuMhm^$zdAM)bRoQgyf-tWjFKr9W7$!&k+JEdg?}TLR{2e*CY98B-EI@dqe_7 z_^S?9S?ZR?+q}FALQ98I!3L%mJzm%)?6@jZFS7OoD6+nJBwU<+8)EiSk+k_lO_6*e z&RZN!lCvsttyBgZ-nZ?)sB;k*X@1i8dNpkn0Tt^hro5r(QA2$ve^TfD>NE0X!hyM+ z)7hKp92fAc0vbLWsTK(*KA|H~xo(|5FiIX-`Cu}DXe7+s#sZr@B4W037TmUgpY#a3 zpX;Q|s7dkN1vRvVeZf_a;(&&FRgOqGDE{ZjTljJss_J}V%?+Q?2Ps~VH?7}sWSV;5 zSSZHjNpK=nArm&bpYhaa4yFPl6unY;Ym8^98&;uB%>`baR+{7aW_fN#b{pH&%KEUf0`;Vm;`#5*vkgNQxkKX7~Q zy<$nL+NgX}4|uT6%E0cE!F#>Su4^zO{=O zU;2);9{Vgm3<72=yU2Rqda}^g5-h0j_!GkQf@~jdoG=Pl_rAC#U`4vDg`@i!rJf)V$ZF< zoj13DfJc-4$UgNc2}h7z?pW*<_Y~2kw;Y-ln3fn;Q}qb1*np>c*1!v$_iFT-@>w_;{ znIbhJ-}+<`4o2W(qr0gLjIMjQ%!dTn&CM)*vVhnMIDfRzfClvB>nrRESYxwByS;%I zXPQjke{1jT+-g@){ZYq&_1sqIm&!HcN{`wZb5n5wnoY6c7tp4T+q*mXGMTOuCU0Ir zKuSt2O12jV?$7wo&AmIdx3f8QFNJ1#Slj+epGU0HTXC1F(ZcGg#tL_LMaTKokPs2C zmL1S)Q&7yqavT1PP_oWMCd##sR2+Tx+$vm03G8mpg+2HDFnVxcYk=nV#pToLXR_&; zyDjwrjyNW^I=Mmcc)_WjV1p{Bd!qZb0;01@hCRy(oq%4*dx?;;Zbrwn;<=as+lQAl zHBCKvBhSs}1s3#Qk)4ujDG!T=ZftJW9L%~x2C@tu?Nk`YahxRMJ40YB_@U?Na|5}G zP@1M~G0zD1t&8o^An>!S{ImB6E>%^zS;opJ?~r1r-R|*^X4fFlnv4F}8C(dBjH$ky zkf0V(Qcv5Pqh}jB2$-caE6jygWSNv{rb6SM2M6QN@t;t5K5Mmn^4z(1faJrDs$6U- z7CwI>dQHEdZlSTyz~uR}IQY|xZ}VN6M%M?ocw+HaG$bP}sdOtG__Yg+Llu|`oJxzS zr!MEo4L;JhcV(@W0|MvDG;5SYm8rtD_Ds)o1QX-$5Vq|tzwiO=f$JBwY*XIf*z$AU zzM%_q1!12mO)B=D4JK3`-q^g!_n3O2&~#Wc|1G}f26%saO{mBe%VuLCuvvqt9RC57 zIxBm^m(Z6%#jwuR;7Qem09f7)hv%lWmE>prx?OA|HX_&yyp*9 zekrr~#xYt>$c<=<^;q+ z+IYA$rIK<=*axH&=<$IuVxdDp3i^UhHHE8;E0{&6MQ&=uJ)gShldtz6eWcU@4IJjb z@ZuoTw}G~JDNLe_*J;7@5wo77V609cFfl8-o}f48QTEDAuB@j$Zety7YuzePr!8Ja~Ywvc){XwY3T(I0pGJbUG z5!d~bpq2G$$w+^-q{|`b)}CLqNAYXw^9TzcEG~tHhSuqC32v@9E&cO`>v z{Xh2JDyoj)YxE2jEI@Gg0|a+>Kez@6?gV!T79c=ycXxMp3GVJ1+}(mt^WE=1bMLHq znYHF^?pyCutE;O^x@zzJ+xy%zxbCUoM5Q3%1es(fPCZb7zpsch+Xn&dGKLkKDce z?F%gL>BM|C4ijz^(aA<2g2}pB{Uk<|M*m4pY{IDCAHurR(~WeQlKB@cTN|CAa@J}#l9OmTWKcq2!K1p( zKl??njkVe1N%8707Rxv5Q7s!`&+XxIf=^z7cYDKiwrk9d{0ZDyRgb_5x!c0EIStO= zGx@BE`wu12{_oKtezQmUqLFs*HY|Zyd^?=PFpZks1A2!*EP*y{_}#%?e#3+r5RKGU zEco-ujJr|3XF6CzOT)(mmWh*nmngr)^|rf)I(D*FhgVjoCeH7-C(?WOY_V$X8%V7H z(KZ*Sk>t(Z&l=MWJ^SNEi?*GPhf?FebtZH~4J~C>P>A~r1kG|~gV9z`inNXsa8}z} z=Y23bs6Yaj51}yS95$~vc*?J&sxE-Z_of+GNVICgyUV*x&bSGBK@egY(+Lp9lPEvA zocEy3hV?tM7ad8UMmz4!(BZXU>H1%OZU`}EEBE$)5{4HdHlx&7}DXkl{bf1 zjI68L)m)J0|XwT(!if3lmenh05T*JVijpm&|cr~6Yd9c(#4_k5J<)K&k-l>ZuX!I9~BwvHO_=5W9y*ErJm%1ilVY>ji{ZLpsHJdwh*wyx^SXYePN zlUk{xHr2-?u#%~?IQy&I4L&lP6;fb1s)>o&pZ?|FGSL0=af^W;b`blnI$~mCn>w+b zswFvHFzak$qmsJv*$qzj+pik#iyt4JLPI0Bi5S0G=zMI<(S^>dg1gEL1kz zc2v_tNfj5b8givtSKk(fjh00skJqTbO^}l}JLtvH6<>{DewFwQz^Ru*=nGoq8`_a` zZznk}!s%alRh0V_ORlRSp~=U@`ld2;1=R9&5`H{hpO4gkB6D_k&(*!f``6+^Z?)Kp zJx8nFKde`1F7b)JakxV}74Kzl;7-`1SrR?or(|z%3LUA!z2U`_Rzj2+03cu$e)*wcZ11!=vqlXU40wG%1voeZ zC$YnP=|h#D9S{i=X^Gum#aWDvnwdCw$YwEXn* z$_z^q1(!CcN;DVF7?3N73uM*W#Y?^=LH^HvxA;ccXkP$8(Vhg{Z-v&v^u*`;tb?=5 z7B+qc{^l`nVHCA|TTYp)jyVbIqo22W&L!EH;}MN<71DxKMP&UIZP8+$Gk-`56|tgj zSAM!)%nBf_kdg#EIEfd}hv8V39Ch?%QsEzhO{Iq;N&bM{#@N1PI5u$phXo*?-Pn+& z8H@O`&=hR6^oT^qzLZE>_wsl&H;auuF@P9Ac}BlbIWfxvv@#?R_Yc^wj#7kF?@VJXW#`tb5}hE#}&CPGW> zQUK-tWL^`F;H^eM&(}+QIWw=x;arCD|L()~3Wow39Do7=Xs???ksAuPr)xTCH( z<5n-H*@-lHt^?iu@fvDh`y&!v{H>BhtKnRER=OORPB$A-pMfKg!q0f#qtiKQ613CG z*^>A&S$@p}1}i6Bz1Eahwi^s%G^xZpVba1lT=8zgakwSxrMu3UPmT5-}#F&>4Hh@S6IL0+nuot3M2V&IbI{6nqal z`!|!mwKcJ^1FGg5y2>RkYdPMH=^JNQ7CN(I_{otGM5rkmOLX6>IqZ~`R=;qN;G)IB zNr=NM{aoOmZI=;Mu}lr0r;uL!dN?_hWh!H(c5^Y9oiyuuj#jH2^TKnwUa9Ty5At_| zhOaT4A{ovl{+FYgv(ZV2YZmmF{y6JduBCdDkzcXRrKPMyb=Nme-?vIbv907zPTnk< zDecNyWNBPT4q1Um9wDf#ecv}ew}}rcJ57+PsqL(otV@A{L~?HrMA9_*ae4X-+B$=^Cv zPkFXjGBP*-RElRCEFr?TuHBh&v$uzYZp2z^37{%7%P;|HQ+ZHrkTC*6KovuFb7KwP zra2w)SM0ayoNO%oPA#mf94Y)a0R);rRH) zuSEy?Iwcro(s8p%vBl@>WE3X`+S5uta0xVeBP7*33*trN{gV>-8_YSxmaAOVSWFb%?1S>;-9lH+Ev>CEtdt z$=~Pk_g~!#jcNNB!vJ)6Z=&KnBk$rR#(z*c*-^f*u&^~_mT%n2vPYm;uysCKaDXfg zv#%^=%nORtXE=?LNULmc9C9|(w=ZZNu@IGj?UQ7DyoE7(m`J+TWV3|mc=C658!TET zuzaQ^f|Q9fRY{&DeF(#!*FkyjC8RIjGH2;vu1>* zKfa4tQ^pnYv{~0e)1Y^eaK*R{nI;QcXd3!m#2GWL8K=$S6jG*o7l{{kd?IMF)P(>G zDInqv88_&D$#NB$Q|)R`TBWZJAmA!Qh#8H5prEIPIrlUgLyP(UNsds(@2wh<24 zqcD7dr87XDG`c|hj%H^50>nzgiFg7s7$;Q-KS*WWFQ{QoY=%Qt?P^_reR?HouNU}@VHrQ-16dFV$9B@jY4wXZ z^FR4yKUQ1|%OD?ALjHDFL5sTKEEr-XR0H7N9zV$f052^7*xn6^xMxxrCk=oRMn)k@ zmPa^91t?KavTXl+h-wQoSp3oHLq$P>M#(WnZikYe&T6~QPAeS^8d1LJd8MIix*!uWlb`x#sy zmuh%4TU=FF*ITaD*wk5Fs$QvkadTsADFrl?*e}|O4CdysU<3u!pitv>y~pKxM$q_U zY`q3>mm5U*I$Pu<5yOo-)Qst@$vNB{|kMDxyxrepr`KF*mqI?@J~7ugDs7fKM2 zXvC5MK>I(b0<+5(nXeg0+Vj7E{r_)2h(b)?ykTd)P<0xAxO!ZN>@pWX{=6iHfCI!K z0er_Bu}J3&imU?(^XC^NeOm#2T;)$um3bd*F_f2Jl;|36XvsdgcR z3*c5L<=1a9q>mH17vw5nUldHHUVLQ%nVcomHKe=`6F}S*N%sHsS>+2AWC_IhCj{Zw zN}U+Fghk_57VzI+=9n#CAWq_6?ti2DYB%uG&+y_k{uoSI>gx_T+zB8 zeZ*PDz6R9qJ15rrbDc$_8v9CpSDUkES4X#g_*KJFJo}9@dRdwR4KZ2O5HeXd4iV%l zMD{tPCV)(-MF=E|1{c80GN>z6Q$q}<*n&kXKFMa&!df?Zdv)qS_+v?ONeH}i$M5>H zCFkCe(uA(rrsnEoDc)pIp^Lm)Luhfhbx3|}?CA8+(CF&=9{?=JqfAI(;@}J%oW&qB zGJ+G&c^iu~j*GKO4Yfm?viK+_ZX)_Ah`pEKg9mG$GUu`vf~atgx9NPO(#*Rt=n z|2ZRy{`VOXej|ovD7yRBqpcTv>9DwSTA#g zxC%|b2Yu8LwzSM_ke*i%?ajMDY~9r9&NJjs%gj}r?r7KJiBVFvIf}%jhbO__qdH%k zT1h*_1qA2sj^FI`x~2cR`)%2y0YtPdypO8H;E_D>KwEoCs)p7Z!}xI7zH({AKSTuu zEk_U7bbFxth@uyI&~k0o+1Odwk{R}8 zcT+Jrw7y0YTxRta9n=?2!)9%x-9IxUyO9?5330^J{-dkBc>8deZbgh+*t-gBc`>uBPc)2K*nWKltD_*JT2n!n81)$lv0z*t; zk`H=vUT%MLP-=Zw2oZph6h~5eofibc1Wa9|rG<7~PV@7RXW4fv^Hb_i#>ViMA|r`> zgY$2nv5w~z7^H3cAVH!)SVHj^(&5e)AmN!ad>~XmRsuU;iyQ82Sy9!Ler^5*KPfXX zjvmzKTfC+2eB#GKEbrkw|6sAc((z2^N;(_K%!rPkeWhRHadA-1NAjvbxBsVTW!3BF zWUr}Ki2UczecbKWQXv*6;I6oLg?YWDd|UXzmICqH4%dIZ5=M`x8?`b*|KDZ z(*j?1;#d;Q0qi>$C2;p@6F(?a+W$XC+GOF}nq~FBNs*0-bdm%oQ@Mp*OYJ3hNR!n> zMr1^T-F~jZ-wl^*Pv;3+G1>KKg~)I)`3RsX#Cz&~jHF*&A5ssEuIs*=9kO#QFY$6k z+Xw#J^59P?(Y|7{j$0OVhzAQ0_cJ|0fPYptv~qMaV@(s$cU$l7khY2nhnoL0u(r1D z8p>IyCmXga_^AXJd?b9um>$!f&%LR>P#45D$PnIMm5?%F>RcWykIfFXmw;DZj%B=0 zE>BGloL|hj(o*pl@bmX7WW}UdHmKPl6D_Q@P@cH@ZY-UJgB; zq7Bq%GBO2;sW4KS8nrtGYMH#*0aI4A#iqETvK*(}ECM0lUw={Y1;#aAXP0@gr{9aF zy}(f|cp`J%qQgZLAFzn%tJW!!f5DU!be=^f>ikeaW-yc@g9XasnB5I#iMwZ zU?H{KtUOx_C_pD^7nM|0uFME&aB+{?h@Z~Lm>UMxT={oGn!+?lMn)Z3FGqg5w!W5d zz1_-UEC61Q!~ZNUgC2+RRhoZS{_zgi(8}b&46xxHfP>sY^mXy2rE0yqd4&;{_4ZFK zZ2=*lfVVc4tl-K5#SR%CT}hMZZ!pb<9~*mT7Fp=DwsJL1?@7u!o(_K9hg(&p&A;il-GA~b9`R+<4Rzw~8d{e!%imbcH>aPS#&WUK(;;T+w zXma}S<+qk8vN(U>yc{1#uZ87tcQWM6LPLXac6OG;VuH{gj$c@H-NGWuvw4iJGt;Ur zwNkT-aTkt&CFh%`&S+2deSd``wXqcg9_$&Y1i-hIx}E;X$=uRBi2&D0m*T)zTu%Q* zr`1JHt@EcB0hir}bY9oJw)^Yro_JbyyZv#(q%W5bL^>a)QHjqx=Evnj`CyR?7M?XD zZjXH}jb_$6@Rw}NnC_9d^34E+P)Xa9?ueuO!B?iY+}ET3CW|`EgGE$ z3uj{`U_&10=9c;UtyQ4n#RIcP5~;s)|LpI%$@L}A6(-iGW?qcWJ9xd9j(@P)A>(_y zt6670g$#HF2I(C)uefgrSC9R8%0{Dr%8P`Js$4p4w1~4|$9w7x^COq%KgI!6)4^0O zDh7ssAPVu;a=qpE^)~MDOhJ^7CFU~)_MJ;hOQa+u9|Qyh($YyuNj2JjFjwexM%b*i z7#vRLSJl^l<>^<&iy~9y**(WsYj(z;K%29~$i^%w34uI6TkDHeq3Y56Xj4+j7A3&R z8WJArYgWrxZLRZpwN$llM4hq9@?73(y&Cn}jKSZvalK@*%q|`9Aa?azA6FZ0;>}jB zd>GB3pvAW5NaJ;YSIFS^eRJ&q_~D>bVUy5WrD8*qfRzcYi{3+{f7W+wX}p?tcm}2e zncfp(nwm7P{I1v$$me>iX`N13FTZaj0LSaJo@6gPE{p}5sn51O3ed|Yn zbFI%d|H4c{K}D4-9Z!S9VTI#zz8-EmmSONpb`~c5QD|K#|#@z zR)egnaR4n+va2YViIr7U_Jmh9lOHp&Lf2n%?kI`N9ic?shz7d-zm9j?e%unIFS~9+|9hC~D~AFyrEI$!qDv@9=y6W)iMYEzIgjV= zyhz?pPSw*xzQ~IjlROqg?a)&kW^v$)IygHAfI;2Y`xD=6VMVe(^SYe)6z&&ZRY`Ks ztE2$qMWbF3_O(>)&&UX3b#=A2&Rmg_=pKg}X}^F0Jl6W8X+rJ`zeMG9j_>884a|INY?#gB(am{-lJnl4YlEpY<9 zR)26Xq1A~wFyn!>9bX(iRNd~)emDcgU%BFZwA!^g-;gIV`Onjch`+mY0Lq5TdS&S% zHC$W`*GmrUdcGmNuY1~DohmGND><@P&b+xWq%D5J--LLgWVu~ZesFXa7D`o-V;CT< zuVS#cz483bn+%935sbti=SSdhvHInm$tQw4!%|iEB~>NafU_dIa6b}G&^7e<&yMR3 zRoHP0l5Z>kdLhnyJFIJSbuK<<&*pTarF?T$F~-5e+q&57`M%OXkKI3# z$k?ol?jai*8VcpXi;agD0@$q_-`YLy&ukm2V_L}XTbmDYWLRcW6w>HETUaKl;^1Xg zat7uRq8iBll{JcXNB=C?DyL9YAS~XE-9bGS%?)iErqHy9x33 zm}BsIkn)tY8e;d?Rk@)KOfFUU)|RQ{k6!khgM0Ffjc^)8@JV!?((xl$MqxV-u<<}( zt=z!ZGI>raa2tWks0N6*W#DD9MIJ)9FR|3XR|nLXAW(ujVA(RTwhjlNXGeZYGc&W% zG;VYNMfkqf!cIm;rf+1_10RfJL(JH_tqNv&J6t-(OYYB~^w zAHj+)ARbsYy*B@VI0Gx~-SBl?{;9rZ+K)H5y`+oPuK4m9@V$za(FcN~e#e_khgE*J z{xlk2-@n%PCEp|^6I-**##4U>8x`2)#1NHXe{BsW1U?IY%7z>rUtgbhS?f*~F|w&D zH=SpX8Htv(TW@KYHl9$<4ZCluR~Hk@L|Q!Qfux2J;T}rT|I8H_-;91X5uyH837pu2 zqrS&cK5J0KAibo|2Sx{o&=#3&dBbaEvw;!gCbshLQF=wtJth7Z<^nTU6Pg=Agh^-0U4;2=?9~crPU6mtnc>SbCuLFCBGq+L-d*8BFx)H^<2?0yZd*)l9*lJ8Ywg)2J+woS$+GFS57SpATxy z^=$1U{togLZx7OQ?SN26KyP7O2fGR{EiGasvEUxj294UY-g!h{*T+s)7M3PXrxV3J zH?UdTD z2$7$T4s+8$VKL7fk8^)nG&ytCQ>1u^TtevjYEy*S{>0-?(!8AEAd{30zC4cdWdr?8 z;qPlXNZ8mOvGd*<#8sR!C;HcMZ3|~YxU7x@+&|$uWpN%lQ$0>p!|*s#T7z3#xE&(} z%I;NX8Ynp`mypqoV);x=uA>s3U#NgMjM(dYEWx_1^|o*E^;>(SNCZof_)zcJA;wWs z1636}Wc^{_4|Q&|oQ{@=`Xwab1YDv=BHO1*H}(DMM>umF11q?ht1hbwtO+Mu&Ggt@ z{$+TGsz*h2u%jGm#g7amO*i`Al;b$O>HZS2K=OI9l^4FuAX>aPrB5VsmZMRpmW?(R zxEJ;0vv7Js^=ftcv#KZYM@Wy@!1XL=g|YMR0)VDl|6`H|b9$k4Pmg|j6Y!4ccfRT% ziDqY=hoU{7{ixs<(L*+kMjxFhBl0bm=Rb+d)9>@$C zKu&7Lyf-1Pt?lx)#7-oI+Yq`K`_s{YC!X@daaS6zIj|dT_RO@rA}`F(oW*}LWKdO ztb)vlTQA)wD7OqPZR+CSD4ear4@uTPClq9&a`nwft`rhX&`K$%V_#t%FIH$iq z^-X5*U*3=F0AIagT#lx8JuMKFM^PL z(J^@N!M9Pr9P2~*40)DX)um^cjc#QyFM?E7?{>J|Xl`zAtL(>5JYx-iMc+l~c7~3- z86&pfgq(Fo^y|e^=19zy;Bc5B-^1Om_`GV{7Py;O(p3jN~LdCU)A3=+n{o(gS`2NKThH`e-{NiS_GOq0I4g2md%; z4n$Vgu(JhhbXYfgZ#DlY(9#lD>vw|&{V{(XM`$#N!V3n!L#PsQiz{i*scQO69=HL9 zhrb#{vKvP#ThkZ#n)-~`br~Ff8)A<6cgMd1nmn(uDo(fO!}kbI?8EtWkq(#0H;9x8 z7l{7dKZNov1bvd;ei%Yei#j8^60MhxZ`!!UJm;r-qm#_g%dH)Fjf95{uDL4iPSL_Rj@JwxZava^0xmduLOch*^^+h4*ezrnuZ#YW;OYypZbC) zr7yMvBm58$IPDh-ZSVrIGsgy&VA?wK(1%>fo?0#S{Lg6kqAego(P?99z#-;JaIpbPE{gi`p16OkLG8}ZU4ok(bV`VMkudg>zMYlr;4#vl1J~(_ zqB~=@;|8t8>-h2BRF>!W%>!TFzdXzNteVlkN&gJPTk)zIvxX;E$RbNmHzeK5o&c7D!9cqq>bF=b_=olFB}facy%>6$lqILbEDy z|IaYJ=3U&r>}j`nUb7hx1w0Y(@1i9^PKk?ZoM2$ARv*!JX=_1rz;m4XUlCod_eyb$_vVv| z?qy3&F)KFy^d`|Z4FyG4!!amBU@U333GS%{F9&XqC-a8%))L*soQuA%N3=vA31oh_ z-rqs$sz1?|^aesbS9^}3*$jPLVOc0i!_;oM@iFp~z_|*=ZWKy9Esxvapu(2d-Sr2p z{ZW1#+*|Va@Eq~D)ifccS`p_;?dv31nET|XSg58Wj;XTe36@~V9d_LEE`52yLn4lL z)pVSS$t?{L{(JQM)}C0fx9O)NMFUHRm(lSif!18lrq@98OMF3CYDBEruO@3tVnMI4 zKF37Nrm>+|Ek%iPCnOE#Jf}GRVr0^9}~xgc~gJJljjH5NU48G1tG;+BCgi`yi}&pnCT3)YmWQ z(NQ*bm##-F`UsqNj;N(vQ;hMqYw7j9Qy*!G_yL zzRuyRZF&CPP5WcnFtVGok?6qBXu<_NXb#fL*s%S$Mimu;HYCJYVi-3jR0NkMjH)UI?d-QV$;}ab(x&v*s?J0o z2l9eXUC*xfbilJJTb*}6udv2b^hN!Fs)(~aQ0#dOG_C89!_pV$+V9H`m8$INCgO5C z|Ni{2+jav~uj;S)FiZg=|LKNBS^uH&>`o;3Z^U*LDx{yM0F`P|-N7m&JgCzy+z%!7n1V+XJ9=p?xDz4T-r83P{8b#svN^QSZF`kL{3R73J-Gs*S}m zHc}ju9W%mjGaD_sXCgTn%74BJu{^3>t43^0+?=kFv(h?xr>_%38Uwh% z`dq!d>ik>W&rV%V`M+4kA)up49eRHiVTq}CqgikhwyCg&7TJupuwDk!a7_!j3fEsB zEziXb{$&N6iEC*n|I(>t>1kBKAc{$B6T}&w_>0x941P;<{DY54E6ec()DU`Vleyav z=Np2aQ7Or3>&>~56KmaS+1&woL&`ZBN(QNg4jJBgT8-0WGPSwO`A8NktxcK-sne1& z%G+CZ3mcr2v=qB8%7AKVBN>s0k$)#G22S}x5u>D(Qs|(E&SEtQuwcEHbLr`QYPa}D zw61kmp$s%z63d;!zM4G0B}O!D?=sZV(xhd3FyfNB@5L(Xf0WeO$rAa87d8zJCnhpSCa32+K(9#`jw@cT3dWs}D`qqSQL{?Y zpj=x{0cU>lD)oCTcZq(PT8br?_*9OW^1q8GY{N^17@0!8Y4J${~@i7zS*r;)33Y<35=Z5m!GJ|FnNNTPqOWb<2X zxu0id!;8;|o9)i4xrLQ^S~fNPY7Om9NlBqLS@-WpuY|x*M29yc!|TH}OAFc5Ls(?6 zW!{Fgim|>lHRx*`+7LK;OzrlX=Vm*SjL^EgqD6viqO9&_Z`s`)4gFZ!FqY=)n3$Rx zQym>`qsY&&pMiNrUwQ?-@UY*mn$^@aKLRhxP(Vp}2K?!T!1Bw*CaBruHQYa54P5d* zL!{XCZihH0=jt6*~R6cx!#R_;b@TqV1K<5gl>jm=|U^*=J@@gR_kEKUjNt z!8tpV&d&c%P=ten_7VF97`nGcWr4I}(a9+xVLQggAPQ1;0S11GJum62Cmdl-!(ZT{ z+-tQ`)#9MA*;yfDr6IA~uV>)YsT3XB3F8CoToB20P+T~gX>VN;1m zFZ+faC70hjx=M-8Uw&0hBT6L$gP|;jp3m1I4ydCWJW4P>)!|90bK}4gG?OjVV)5iz zEF#L3z}Tx+q^oq%hzRBT=Y3Kp!z~7n6fk{_@fn6M2EErY#33(S`WYe~Ls<$OY@8=j zo+9$~8zc=aNL(5aYfe)k)tMn_=t1J|rH1q=-CxZiX&6Cn@1@h$slU{PAuUKjZttbP z%9BOZ+#oH;K?3hJKe))jC@Mi(P=Ex0<)=KAAZt(x15uBzZ2vBXa>xZJ8bDf5=jFWL z+g+X{qK*$~L7Uh8E^?eEBjjvB&uW5e-o-8Yq;7RuNDIciN-028vrb0H{Rqt_0)BWG zKb9wkWVt}INr0R4LFM8Q(4>O&$J==>_0lk8kTlXlx&Colz<^*{S7S72qEB*>Rxh7i zi7E_LD-ws(wQ4q+(xeTePtK+)Lkg*kR9uB8AY^qwv)vYeNQ-S#3u20yAkbnE)QJM* zp=ZNCz)P)J#0`r=t0n=L0kn}KRFFp!KK^+&?9#iWkTkNuWp}(}HadhedDgs03>b7` ztr?B%E0i^F6#^Z=KnCeRnKjQz2ro5W9vDLf$%n3JNou{(p~Irz_2GwyN6*LzEvycZ zoBeV+A7gVaZ40w8IRf1yHVj%+du0U9O9($bPn zyD7>aBubvG66BKuSJWQ0H<5{gh=@2_tO9GfQ1K!HzP!A|;df_yetBuWE|5+B1(1rq z?%)uMUvVg?|A-;#v8QvyL#xathyaieC6Pf#B$iU{>LyCV_8nye@(%(3JIwNqGG=9w z04|qOcKjCjARiVq#s5JvlCHITU~l(__Z-d?q7o4y-{A}pEfvV8BN!PO$qNzkI1L{1 zjS6l`p#=V zva72r7Z+DmHK^V+H!M8bkPOnoYsQNK#Gz0FIAFNEU->xSImYIurmodhf{_GzQDtSc zlhadF0)kJHl9KhNqH>vVU&tZqI}Qb~(Mo-IUC-=nSMx+#eq56AI3M*61rdt={`3b( z5T;b3O8q}SWQa$Fm!2L;A%jo2*CExh#OHC;|xrQIugI1jBu&0VZId5={*u zKn$8K9+-eCmKnox&~Z_~B|L3_2pMFGG%x{+*%fvvA*I2s? ze-MeQ`uRQvO3-Xl;MzAmk!QlV4s@LlNhAT>#`!vk#HZr zen|41*HTVs$>s|3=|`OekycA`%*#AJN8P2cRa_N!i3u&ieUFK{I;N=ZG{=K7vGl`0 zK^-K`GUwdhZwDO_PK^mc#{p9W{_t8ihB84fIE6Ss1o{eX5&#vKWpdU5P*F^12=04M z)Z}hpMvPIm0Nytj3QKPcWYj@Xp>3Pt`t5sY84`1PACjFy6R)%){?8C(S$xoJ(qLs= zsms{7xI$88keE=euA7XBYX2~_91nZf?l$KIDGd>bq6Ia`=j!uh5YhBLOUxD(HMN3h z9yrKH;wB0q230E9<^x%lC-iJf;0--)bX;7eu(DW+oRD<@s5A^Lo#F+hco-k*qlr*l z#X?3+t)83roh{L48NRp(5Y5lzgecUBfL$qXYd9z`j}Ix9Y)bZh&+0H1Q=T5h_mwUu zfSwfrdyW951mUqclYIZR-wgqx8z)qJ(++%59pLdLk5L}tR16WSg)1tH!k{3Ji7{RK zthyXSkrgANTnDs+Vg*d?I!2o1AVNb8w32PVAS)(;03seuSqi9H)d4BqVg3^&4LQgp z8dXY))#ek8$ zZC8}hr=6WIF9y;N$$t>0kaB4V5KFU8qsK~2Fb;JliH39Z2Yn8L(d_<2pUosrGAsT0 zyLo$}(3Nsp#wCeX+JjIUy<1#5Y4gJAU^IpaO!lmZvdepq)_*J2)%LBZxP@+6F0Z?F z;sN)5yRDsO9X`avrNT{0mH9L}c6%c?cH=BYc7TH;q@_nJqN4*>Uk{vKUKa#oNVMpF z1MbJI8zyGO8pmDRDuO)s=YB}6sRlCwU2zwZf*-AQ=hL{pO{Elzpw-8s*% zj2k44XYJ0P!X@&lDC(K2D+r9tZEa0WRpewQdh>b_VzG!L7YC`f?(fg~`r3ckRM4WO zE6B)dt_%%$N9On`^N<#o09A$9w3Ni^`DnR%ctsEOj>Z|gDsh)$ls?{$(Ob;=D@sZp zB9M*`c0B>8A|@(o>7+gJY}B*G%bM9tQcR_CO<-X2IMME;i^Zw3)30X}-~ zGrrVPi#)=`QV&Ez^4$C=@JShmD8=;%%#@(~X|f9D?s%O*`;{>{D) zkQFJIi5CGMQ3ChNS}UB&^Q^7pr%X=%Bepj+ouQ%EG*`E?n^b8OaWHJLV^Aq7_o6;F z>YlK^e-PZP4WnQHImZJla>BIa**xIWt!5e#v0(Nz%255Q_*>Yg*jbisz8jbuu2k0sbQBj~eH?K# zF1EM-fr0b+g_(zkl%LvO$ZUf=F*H>?7})uM<@&DDtsjg2b|0Gn*8M*n*!`0$B)lUU z+q|pr6S>~-umNG;N#@V=*6k+8H(NVuWFpgS!1NB85sN?9Y$y{oHg{2NtrNEPz*Y}G z!o*a?FKcv~*f{qxmA+cbJ6L28*U8ASBQ<{?=V7XZ95IZ=S~GfUsI?D>t#}_B(s>f8 zrSXnD$9lFQn3++C^NcAAUC$1-9UH4mz7|EX=|}*U2@Zu_9>Kw}`f#Bye1XfzM82KkPj$k z<59namjBVzEM3Fmb>eY{rm;Au==p&rVf!f%1$6I;EFvS>2xRVg@-#uxBL#dayLyBm zif^grLm93XK9Y>~n*HlM?fm|Ljwa24gE`E9XAZJCU!k3!rk-nk8GjL|@n>))jSlP3 zMFMV1F&|2IY)h2UqW^2WCg25A0Lm3n7h9bHuP)bW#URz4J59dbG@?b7Mu80Sx&E{@1B^73vQ#G8(d(Xv@+Ypy>OdpuV5R8R`G+HLy%$7BN&ug>u9EGl;c z2C!d_NF4?R_s4KqiJVUBxDMS;bO&X@!^L8C(m%6kPA%*8d3gg7WlmfQDsTUa3?gZG z8R|Tw=s1jD52eY_D7ajrTI#l8+mi-+6vW}~;-4GLjbu$NmN(?rSN3+*S#&+X0ntjW z#9EshRFMvbHN}R0#JsIr3#dklC2~K$XHdT3ChdHm z6)StTnB9Q=BAl-;jB`W?3~Od+>_(^LV*f2#wU9Dzr`*O5bY%nM-Yuu2=}-Kvt+@cA z0R|FnM|X)XHmu;w(=XfQMD1T*%^?~an`UXwoxkHrKMA?-5QB|POmZ~&E&rYId~NZ%^C{GBeS_&P9^bngMY8l z=gz0)V$*%<(fD_@f&4Y84h@p%7$Gcx=mYKc$`=d7pkw8p>|3x7CMqGuB#yu(R?sgsrkQ->oOQbqL z6$2L&ln3OwUF0bHWqs`Oj;iQKI*We%B-$v?pwVo@jfReHXE#8a*8Vxnm;7n--u(I^ zUCvoHNdU`hm2YaJ2N5Y{p&MXqe5#|F_vl{<`E^#At^Nk_DZXm9LjySvx+;4H91d?B z?Ds3!N9xmYsl`M0{xdK9ITkMGO5>xyDFufU z**@dr20sOQZZxb_->aYie@k}9OP&cAN|XD`zxP$QIRVqPcB2j5cd`_Fv-9ao;&p6c zu1$rbG{Wqyo6nBxikc{R4iyI$DqrLkA%wx45&Sj@#j0|r;yS8JnCf&P*u zRA~DH`b{)mU#MIkf`ewL2@0`@QGs9j6%W8Rb zUZlx=-uYviD26Y9&Eh!-a=TKc)F#_f$ya&WVzDZ_NjYEo;{JXjm=!>FU0hs}n2kPC z%B6rDpG&4id3Fyk8zNFYzHLj-vm}Pg(-Hm8mj5x_FnVQ_;->dDuH0Mojk>9zx*cP_ zE;S*z60dId8!le{o$}&pZTQp4nbEUW3Y)^k(U+7tVBc9Se0k42Riu%PhK9DS)S(Si zCCNZZRO*0P<#34?a38g}s49GUjB$C*84@#HZ@B;p$7AoF%9WhTm%)x$^MQb}+a1PS z?eM}62-d^&$2i!WS2s3zZS2V+y-Df(5kY-3x4E8b6tf{7c~-R%`1DabJD@i(EKlTU zH5=dk^`O7~vIWfrzIIPx;Q3qVU%(sYd}Gr89~R(jwRJ2wFr13rVlfGS_2i3rGmS%9j1Y`K*b zynl5Yr#^@-FG%Mf85SZi$Ns5A3r*~{vqmEHZv&tzk|Vm zCBW`)7X+XC4IB^|DGCHd+HH2jFe#0tu!la4_4Y!eQ_1@W2Et-zXJ!(sRcHraT-Y@= zHR<0T%`tIsFgW4UB4_4kd-RX6p0qZc63|)BF%0iI$uYx;ZM6GQnaKxo11=BM@1&O$ zg3D;q=D)z@-IfF$VTMkXQ#Lx)|pTGIfjnE7jGar=4(y2rEN&~?w_*( zB+XNOgJ(ZH)9KTp#{lKQ;RAWMjrP;l+*|8GVd7Os5q~r9gJmxbFKDSzcbA{@*TNIAmmr)8%>t0|Vcer~97S zS&Z$0$o-%Af4-}b8BU0v1HRPDW= z-}BP#-JMHe=YEl3{pvKo|17=L+nFDRq$Kb3gNBBtYre@YZEiwZO`5)hMyWOg#jbsL z?I|tsv}aa9_!cmZix|;Ejh+OkrhAOjt}p zrlLPJGu@`wQ19!iM29@N^ax;HC<7G0yz;IIu;Z_^d8*Y};L%V!RaE8!GV`wKX_Y)q z6-6OA)Qc9YnL_(l_L>XPNWy>4!(A^Bo`IFnwHiy4E^AKg=qS5-99J;iUvN#TX(bMJ z_U^T=pz%6O&9}q<1FcEkvN9smrxu+!?Xhm;d(wVuM1>Vfz0EK>+#Ns_N{I+#*;@Xx zA9Hq$wp?Wh1UDQ=>2}V~bV>O41 zxuP?rp5{vEG4$QIc_oLS&!g^`HjE$krVjb-w=_81^Pe<(S(>q>g7f+S?uV&+%Y#r{yoG&z5e;f@6w5c9I<_*yp1;M3^ z4R}cGYO>WePj~t4_OqerlU_doZpU3#0Nzqu-OWTp1JEFJ zoitwGr9K!I`%7~^9%53dGkV+6WP+K~g9r5$W`9a%ybtsTmMX#wo9$xZt>Ymcceobj z9rbTH#U)fV<28TkCWz4Iu!)a#wS;~0y&WLW$%?CHImLE(Poj7i2SLIxK}0I#fVyhu zf$MR1(_2Zd)pDEUD16wQW`onV7i2|8aci^}6o36Xqa~N>dWB^Dk3!~1zGvSCSxzXI zRmYbL+!jIax6Kj6T`^eR_I8T#6i>Hoix&cnKQ@9oz07$s8S*s8)S_e%s=S)58?O?|jNQ z%No{>T{5ihI9A7z$mU2KSE?~QA{;HN^I$k`p{5R$n}Y6YbQSj!_aMbK$&?v#O5oJT zri9q=Q&Lmb`T4L5PNy0pyQ31Fe+oE@ZP+9YCc$V_o81_BSRc`rt)mcfs>8L%9_`L8 z&a5a`KvwD-5h_hOOZHd&8-l0y!yU~iXX2qMzKh{|2QlogB-@Z{!@f;z6CGbe5Nte^BI^1hj<)|7lifnt# zhAoR(Mk&&S zR`XfYoSeLjJVTCyg}ql8hDuh+5J7KU7()t-hl>Rk#*|iE%hdU81iHP$1f67@e_fsA78Z&C}@EQ!BVJ(jI_xh>H z?E$dp+H&P9nhXG}N||dxgqKa}E^s|(vEKj#qZR`IgWNgV@NbpE$?oy-Wx)<+Nt5Aj zb;FWssb4>w(_8d>xJl-xs@9wTclx4vo5VRqh>+Cn0rDJZCB{+Y!CX$iUmL4oM>Ncr)(zr4FE0w%{X77*L z9Nxw9ftnmH$=_N$*PQ$qsqwnp!m8H^w|qRM-ovUVjbFO3=;b7kLgFPtsJs6IWFBNS zELt^zyhURgK-_8i-w%ENaEr8CU)Y`%W`;MMaIs6GRwsx4F#&=U5^wp5+><@rizK}4 zXtcp)oxXlLf&)J&&hdlyboyeO#2H@*1(j=W?rKeLYtG`FodJ@AJyv}Z-)E_eTq=98 zBH7Z}rSK9l|0(+p_l1L_{!|mn(an8UofN7p82+0q>4paGDBw(+@VYs`+q>Ke@O~9x zaFyh(r*9te38=8(wa@QGe(gGuY(@LI7_F(x^6JPD@AdkBh^8Qmao3)uz{^F2_+&z&)Hh zESlKQiUd6}thv zrdIQM`j?w`1YM<{43kqgX>x$d=R8ritiZqnD7mo>yT;i8*B+kaLjsI5fjwt^&L^?C z9d=~Ysf7B%G^jt=u)An%E%l^>u+jx@Z>)rRjjx^oIUrE&n{XW(Zxor4BR|s#t@!>` z%L}e2Srcp#jHo$!c1xyRt-6!MumRUs=dnjh*u3Fqn91r{Tn^zWINJ4X)%xW&9|pn_ zqR(;O260DSSggDa%)P91T7ibORD%Z~a&9uF`5L%QCZ-CPhkUrzLI9Yw&TMw-aJ@k& zN#`fvpZ81}s`a>ktJUU;d_`n!u3ga&%7Lx3SGEoUD6!Dp56s6a>8x~G;;YT( zBU?Xa_C>a`xc__uOUO46zA&uKjyz+n)0dRH-ypM^z&OjTB+O4)XBA#*OR)I-8^4*Z znue590Q)!1Fg?riBFOFLhV1&LmzW}VNMcrJ9jk;(j%ZhE5>Zba_H{#i%|EXS;oB$lc^lx@Jeq8}We1zfjx7 z?0Ij~{s_Rt0^L1^YUOD=yn8q;+^#TzuhZLFx?$Dk21EAsJ-^B)w-z$&+=U)6E!_mC zyxX6f)Bi0@a)9!NQ66l;XSP(5=;}}T)nmmx{`N5qKw6&)5a=pDRZhg7S^Q?-C1u4x zUm$F67B{i_SyUK|XFQS~$Pe$lChS7lX{|MM-XE6^;jKFyZjI1A-QG(%vs!2kSP=H{ z6e7Ku{d9Ef=XDe*iB-}lvzYS{>CLWS*utPFz8pZ%$}Dho9QO-b5oliVE*Q!)b8}kw zrw2W0=(Skrm(`>kgcB|N2}TUFoO2hrV8jveYKFtiB}Z-wH0qAhADc6TN_glCuKz$u zof9*utn7GnAwekqHZ!HsH3CxAZQy{*Nf@M3+Kqa-Z|}EoohI68(xa=^_?7F55V>RY z6LM5q6yi1-#i>ZE+l|SV!;#EVXQd%aQ6#*qZd+%#jJ99}iES{Vt3SKbIC;vg#XK~D zd)470$IAYkgMaB$P{XpO76``c;S2jUDwlhXQKUusHRKELQnLR&SRe04=ccKTcd3o$T8M&}Mzf^mz8sKp51bcljRylyZ%a%#%E(g5sP1m!TYnmqz<`kIYc>q9!rKl2{WbE|b2;abtg-Zf zC|~H5*Qml88ocihmlMy`dadr}S3HA!f9Dx`|3U=tZ2RWVH)U!OVzpn_ET!M@hCd|k zKUQQbg>L=&J-+6qRMI!)Jy??hGyM1klNyB>mJ_w`zC7Dr=N>ija@Usr8g0E5j(MeZ zT{aAyL2d8T zIGHF#dedCKP~Y&=1l;6|8ZWa7{<=BhX^-}B1AZcE2G0KB*S`e2y;jc{O=|^{voF?r z?16wo^7$;)=4Gvmef*EsT|*KWTGeU#6>XRh#o$|Q5!S&>a?g?HSXc0))s&rWsZxR~ zIG~h<8iv!@P^r)$EpfI49_#iDtAvq}TYpm^T1lyK=EUO7;AFvxwE=Qd+G94!?H(4F z-smz6wZ{Vjk)`F%{vLLbT2DB0`*4FR_{pmti&v>xJ~!_NvW$vTG=``dK2Qlp8Y$l(@&1L}#cm@|ygAqjDSzjJYs0vK9xJp zqZaNQ+_WfeJwkF4qhs2_(>?QoJxzt8;0FqBYQ|h#0~eeQ79mbyi~>Vhc^yt+9xi;V zVw&nsVU%>6J;MU!+Q6hUz3N>JDCP0_Ykdjy!c~q)a>OF$E6#GsW%z;bB%S$u{CSCU z$^<6I>yqhXTkvL?N{v8Bli>N_wu*DX*~|0I?s-gMzCWLC(-*d?xuzN{?=ny|HB*J& zu9j6Rak41gn4eKMiT2$df;VGml44XFZ&rUAdh&IbKuv?;F(=^q&us4#dUNm**Pt@t z-e;riaiQ_^M47g`-g-jeq852wM7XD><)6CC#fMylF0TObaxbk}XqpJS{TXz0jPX(> zXK4aKjHafh#eM;|pxN`Nu()1rQ!mR7*7XK4LySb!f;7&ZE%ZpPS zUjmn`)vxlXi_r89sQ>mSvd5lfv)-(tSO>$9`Y}wWthG^njDygkfr`9cPS+>fH;9L) z!qLVe$`T%jMwa$7{le;X02_*d5WdGlLiZs#i|3J1MfmJf zLM0UE=OX>R6bal79@x_4`kx(9kH3a`h#vV{xYZ?5Ime@!Jae%N$<|3ZaHV8`>@{Z<(->b+kw&1YW1!PZ zSR7wAbROPdFr`u!Dk0_(H7uB_ltyg5-_RJ(QD(%n5=MJT2!qJ1yL#pOcszf&s408b zM}7&(!5Na0l%|$?x?vcT89OubHu=M0agsn}@rRQdrVc6|arf^D-@!)}V1oBr;&6&! z)+%~y(O`ZNUK6MlQQ_&8a)R1>#lQTrTru}57)7o5;xMFNv;0@CY7n!3cO5PsOBQok z%G)6D+vG9;E%e4o$6MtY@0jo~Xl+ZwJ{t?S(}Vph(41Z;O1np)X_ayYK_@^pNRF4N ze?Z{F>hPp?e~ND0Szp9@b@yf+q-Q8iGX`t$ILMR&nKjhT4M~3r_<N{dZItow{`&v=~@zy?W-jjirM+z)1Yx|chmcy)UT zNr1?{hPT!HedY-XyP?*wam&AV7o9p-&Q3`T4_{}U`juipL1Ft0IbNvyc%`u^CSO7v z%}&;qOSyDr_!u&>cvWrjaUS9miXh;O|0M2N={Co!7RwabCbIaeQXA*O@fvuRkY=

!gg0o zFzpF`PGGNm9Q90hL-GK!KV(#OhLrBG2K)qdg>$PYUzebPDr&y&L!8#9gh!=>A7>4h_Hw^AE>X{A zIPJtDfA_dDbN@bnjE3VCf$YOOr0nqKMu6(o|S~M>>IG-27gBKk&T!3k3@K3lpSz?hb z^?!q_Pk;JXtUDg^k6W}mr2%H9} z{;fOe&467d$X`_Pu8JNkR>dy2$wN~S=`JU1+zWkA+)9bfVc!3AXt_!E4MY*_YNP8h zGb1ilP`tftHM8tjWn9KW78UcP3PcOk0ggNsXJcz#j1-f&a5^tM=Nkoz1CH{?@-l6i z;5P!Vg4GsBZhck~J14UMoIw!=EEWKu@jy+Fv9#J66o*&CmWPtRV|G*rToHj&lG4SMnEGPA1A2! zsqKS09bNs8EW>7}k9L%~w6&@LHXsW044L4Iy-$MZ`(YbVQb&{@9Wd4vt;p*(X2>PJCe3n0Bjnu8iY~Nr13R0NCm6(i^H&JE2G`8HZ0|3!mcRsHZ+UO&2Vq z?_=>Jy>WR4&<25jM;m%{v~{vT6R^O}BsK6t2A&Tpe5EGV(R)?R=NZR)2L{wWG{$MY zk0Sed$}GZ+zn_rsv{^IY4x<|t^FN{Lr@ybz%pF?ZqDpytJ!u8T)m>4Y_W7qT9G!c2 znw-V}00;o&qot21*G?cEOtl82MV(GqNYrU>=8ZToaC0gi;D~Pe>|fRv7mhIp8Ji;3 z5a&@*OYgXx{Bq?nQmQGB=%;w+7!6?EI!vOq6Z~sH9ABI=DLFB2y1&TFZx{(!j<{=V zQ_eof{qJ%0KPpMY(KV2wwySU>%CW7YB(rLnm5cB&nlIG(4Z(x4GL=Ou6=9#klVCC< zA||lFfCcRul?0qs7UT_fyDqCK^Ih}Z=x*q_walZAT#c0l7cDB0;|*j77pW!*TP?Zo z@MHWyXE4?9;4`2|HxgfI$H94iaK+P(+VSY^>*koS z&ZT2z$;FJOf8m|`jf?MGiSACtqax~5RJAFhOjI=7>@E5EgM%HZDkWlzjL8`oa5ds$ zHOU0U+C>zVG1t=t!b81pwHO$fUIA!34p8;a+1SXK%oSy<2DpfnDmn^at}a9TnB>MM&}NZqVe3Fd{P&8vHI`K9!L_2% zmhaI_2uwnIa;fGFh`^Mf0dL7~LLGH?{lCzWlrnN-3#cYT;-JbQvA3jb6pwA(nrE~h zMeNE8hIlcB)pi4Nh5`DVE-P4(c!eB7kz<`&0MlOV4~YTt5foUaOPE=MX!3YubU$7Q z_L)&SJuI>eMUdj_7lJ+Ki&6r;gwUqHa`X$qZbXou@%g)tBvK*e3rU`-B9mFf2px+R zP5y$N&t1v*7UM$4Vn>s|Y^}tWSu0|Pj>U=Qd)c~MC-bwI6uS0ZG~dhCpVQK0#eC4U zXz1xL@9fc>2U9=`U5k!R^3pVw%Bz_A_JR&_zcfE3=Ya}XpldM|6<+$TsYquQqd?bU zDeAv8ldq)V6l?)Rn^OIyd5SH)R*dxmRnNRMi|eEV6kfsLljS_TH2tQfKFNQC!KcVs zRE(YmJiCzexJz5bW{q4JG-xcU3Y)CwZS;V9`in9^R3X zopwbjlEM7%oB!`=`Pr;C zD+236Y4ef=)Zm42&y4F(DJa!cl4_1gh59B581a#Vxfi%K8Q(JvVDAf&qirEbkiG*l zk9Zm&{ScvoSkhk;Zp^+Lx!nAJgzLD^egAO}S%{*b;|m=8sz^zxW2pnhS3JrBXUDQ&g0xOJC!E-7$;u+#mBlPE z83ad%n6h8%VQ*TzZAcW*&^}05XG`wm;vUFJgy~KIS!nkV1eMZ&8p{RFfeznSWaz;s z7rRO^bV`ASgZ@TXK@Z=-I7|g$nF+Hzn6a90J>CD1uFV#t}{D{dREG!O0URiB5FWsJH zNU^z(gjI#46BLlk-|j$01XCJ>7mSF(OU&oc29fA@Xqra`G(w*Mv=?=6syOIH;V$um z+SLtQQjdv^9Y`e=N6N#42dLXW$;-b3s@qQ`g-e(E$2uss~&}y1vXwfok zc3v0k5vy8i-Lwdj1cQA_OPK&kU;vJ*!9L6kEMkZa9OSPM^TuM z1eU+f+1-T&%BPFY^qV$k2IK}?_KjlS<>le~QF6S03uI}t1jWQ@a%iaD?i{#p*x4eN z|2=sn|9_dhQd$fYnXzxpx&?-f814J3DcWtqBHA)v&ZyVLdjDhy?cjcos{>^6ZnoS7 zmonFlkXgO~l~RqhhuibTRSx%|t@U*ndwY9E&AK;$#IK^hUfjlp1t4Fhl4NZ0!}D~9 z;vBy-jzpNpa~4=0uNNt_W_{aByNm+_1Fd7y5^c$tL<_kELrOr(>|P7}W(!bbHgH)f zl4-ZNpr!IUt{ZhnkckB9J>Ff?elCj9s2+DH#H@Yr<#(T+;QWPBQBka~w@`SV?@H;g z@y^ACkCb|A%`awn<6@M^)8hymJ6vPMp=xweR!XYt2aOz1+qyBOlG5!^cQ!BQX&b(= zu?`I(K-@y3exPi&p)&I>zQL5r!|`yh4@h2XpjS2|pF_8^1xN(=D?>4%E9xgj-KD#C zc8z?Fj@>m}sgCdHEDOBDX-J^*cs}vnNE;J2-QBq|Z+A~GFByn}fWcLDSmFG~lTeX5 z>osAmj&AMD(_;MU&qgA2>%(7&-!YmDgxFE5%b}1TN2zz zr~i*^d?D~SDgmO`SeQsC{?^)ZQB}=GtYS93*aFJ#&E?2jwWHtJuP7cEd#`nqk(+C{ zUE}ywLwF7X*-(a#DKW=DAZ9&@QU9FW0UQp5dtk5ef70!FzWDU-(c%S?^cz}d#*qblcxmA$_HUNqM#&cjBK z^gw&85gH!W{V39Ay>K-9^%LmsB34c&wz6d*NivUwwc?rbk<}Pu&-E(bmjZ5T^{==q zxmjG2n-cJ^baF?TD)}!H!KL(M#U3etod=Ha7v-mnwf^OSMf$6&w7kFb!8%A-WQRjg zSN@AM&7!-#^jY)HIMLO;&5BPpadg<~4*MmQ&0mm6l@$=Z*+ACu^3NY^`CTuQcjRPr zW>gAlArv-?UAdBUzE_Xf7&OI8I3P^%N*A6~-6~opsc7};-#X7GlOpuQQcW5?h$#m% z?vjb-D>60Cwx3g(xr?~Z$GY18DzuD-OqSRcatBb@xMgfRHp`|G0KNh2BM@&g7tDu9 z;Nisu@g{E_g#Ak`3gv z_t6wKy~XWvvDi#*G1kB2kA%5Vh`&jpakckv=Pw_r3Ajcto}1vcd=KfY{pdVi?diLK zMn@U(!0KxMhc_)%a9;{UUrC=4wke(Qck*k~{y)httxuzwP|{!~e9mq?)fqllJUsl4 z;aRn#+v+zFsUL$7rQ+6}qC}7e%2^V-1-%2Cz1r%*;DAiA?IL{Ne<3S-5&=tt-&@Y> zg_*Q7;~t^%`qtiG>90gnRWKR`_Gq@)?ypf;=yH=B|5<18I#o&}W8a&p z_%m_B?}!f+uNC%~5N~iajf6AoBVB)b#dx}ml#bllQVRNA7fPagT*5-vW zQ)gLWGGIW+@BWrewY0a`*?PQ5fubaV4-acl!{C0c%jN10MMqpbJEj72dN_gl+D$Gz z{8Pepo+*0tqJq+EDzMLsuxvA3T$;xHTGEWyxWtQ&|8^|c2vnq8$JkV39t$sipC==) zp$p_Q+*j|L9C`FP_LktqXG+8ts8{|$2zjjv!RodS4r~E{P>UB13jFo*&DRN04-;*~ zKkSET-jz>MfEQ>v>F!me*OIDa0F9^;I6-$iEB$P!245LS6Bm{3iQ zzMV}5O!N&vPMQPlaO;W*touh(kf#60Kk+_HbyrqW#SU_|8+Zi*2?_l|!&DWP&o9vH z+oo;9#r*K%=Y2n^k~;snxNbBl{%|%T!bwFvl|zG3rQ(&C>-SIcm?aeSAV;C78-#Ua z*VhOe10$)9h^eT+KX_ghG#ROay&bkZ2;Pl1SEC5-9pn)uWcMmsuUjAnFw(BC1JS$Y$zC z)avOnzmGH#e%U+YGSV|Kk|$Re{!3>fHjP;@8_iCvg82(8<)zSMx=^;>PsocSA4eY^ zbm7&iAcTYjZk-35d{M0Gfg6}M0hint3kMB@e020R zQK2UKrb{(MkcfFPgpuRoaxqymz$L)#XB-G>t$%&izJ%QNsx{NQU-{<2 z@3O*4pPkJ;y7U>!E1CqhR0lq@cqM@1^mLWWlyum`nyDmM2g>(JBmrMpr<3^P`qa13 zwpb{ChV7f^%ny}EM+W6BV!tW^F1@?K&k($4tl!Mqi|^ZWjY8IR4<4u2fmcCB=WI|0 z7_&*jt0Fpn%#BN_%w?H>i|bRjh&}vt581&gM|*mzp!k4>DPEnx1ab~;w9r8!d?Aj1&nH(>N_N>eZ(heG|$MIr`Gr|+Sz50S7?_wkBJcW9dBc(IA9 z)uIt>VN=;#F=nkdij@>Gc|?IAb_I5>mG~b8W z1d(M7>e1?MSxfAI5lPjA)OkT{LNQ^3FK)l+q~3A2(VAC_3w26UFTi-tPzr zMLmkxk4#lr(t^2JEaE!;)Ia|AQ!RC>Q%7J#Y%v|P`utU-^$lDw6{0W~qa4(jNEP2X z&gJ)QvPufn9XMo!l#p<`Qxp_}U>Y$EX1t5ws1U=s_`KPZ;Hc^BNdD>&Asy)pY@GC; zB(H~~)-LB%2Blra+zfvo7NwHi|ELT!$|cmRXC(8fk5CgtbJ_oXY41Z^@mKDvNn##2|h4o+X{_LXr zDoGh~hAe7xwYAdrmP);@P{>Er5`bux7h2DvooP>GQ^21vXy)S?_~}TmZba{xKYwo| zelH-<$}Q>SKV6{QSvy=4q!g{lmwts{0zT+Rb^8eeLxU||{_Zu?pFc@9Nx0+@u9B$>1Jd z^1{H`R(ez3e7pAhqtbNMv|2*=tHCkh5|}ww-@=XEaDb~zEpxy~-X)x1#YE`DRN**e z^lW^Qf!@57{=u3ee&{;6#X9{h+`t)62eCW1uU4W3uDFmE!gt7A^WE*5;PLydY8?gW z%|twWNx3#Q3jW7I=0xFsZU3+l3%`!aTl+j4XxRt1HS&ma(#7=2xsqb~Soyp3V>*nE zniZl!?%Sd(T5vhJe|xj{l9hd z$&e=YAbg;@{^cQ)j?k55&coJ6I@tX5#aIOSr4-3L!N(7EU^3(T*JA{Sz-Y|JgZy|0B zCv(Nnpc^&c-SMo(`D_PuT^i0gI))&XR?3iIyWN~jjlRXj%o#=R+O|v+QgLbUn_b$Z z9TR3})QcD%S;k{#8ZT0|;h-pO)6S+-n&BKp8XKH&_yWjC90Ty{gkhlME9`DLMWluT z>y3v=`Cp08FVM&A4aGZf+1WREh!2cPxMO&?T{j~O$wQi;P?AY~X<)0i1E0l6;|_3| zo26EeP^t^((D@mf(tc>qXlO3vsMa&0NDc8m=TW?Z+m%5yaHoWV%2$2Akg3FBjmu)J z6*{&tAlph5QbpRnYqh3M(>Cckv*F(z*~FlOnR7%Q>|~MFF#u=(dLcMh_s8q5Sziq5 zv-@x)^b1bfYpIaS&n8~%Hh^!|=<9hBT_Ik56Pp;&Eho4R0tK}(9GO5G!~NJLn)uMt z7lv_xiiCaTz;exuZB{&rgWo_{2AY|8P}s>FX50FX2sc2eJWh=cQd3LibfkV5< zw7x>mm+9q{$(hYA?`5+wBB>{PBK$UHLPRp5=fnF6?(?IPgkMu$uP~`yOnq%ZUHc9T zhdp=XH>0;jrWlAV{z6tp-7sz0Mx{EXX{47|<~FFfTkEX-K@>5PrM=QKld$1~ntAhI zH;eL$iUeRPZ^ANbH}`3rbJZt4U{Xp=^tM4d3}F(hO+s(^ju?GK34fRUj@(8Jh?)Ef zMMuQl|LI^CG2+N`q_?QH?)Bu<)GuZALh~lg9?Ka#9~#WGsk$0Ug0Q@7H1UXnk?C88 zH`%1>p?p`NuxY@0(~s7XH4Kbf(EV=C0Qa{^0zQ9qnp(ePhVOR`NQbG~{vMUMH{@t1 z2Zz4z5PVioeWcz&5x#}`RV`. -For more details on what you can do with apps with custom components and -the flexibility that the Controller and Worker APIs bring, see the :ref:`programming_guide`. - -In version 2.2, the commands for NVIDIA FLARE have been consolidated to be under the ``nvflare`` command for -better ease of use. This includes the FL Simulator, the POC command, ``provision``, and preflight check, all of -which are explained in more detail in their own sections linked below. +For a more in-depth exploration of the capabilities offered by apps with custom workflows and algorithms, +please refer to the :ref:`programming_guide`. .. toctree:: :maxdepth: 1 diff --git a/docs/user_guide/nvflare_cli.rst b/docs/user_guide/nvflare_cli.rst index 360e41ed06..d806a2aedd 100644 --- a/docs/user_guide/nvflare_cli.rst +++ b/docs/user_guide/nvflare_cli.rst @@ -4,9 +4,10 @@ NVFlare CLI ########################### -The commands for NVIDIA FLARE have been consolidated to be under the ``nvflare`` command for -better ease of use. This includes the FL Simulator, the POC command, ``provision``, and preflight check, all of -which are explained in more detail in their own sections: +Various NVIDIA FLARE command line interfaces are available to enhance usability. +These include the FL Simulator, the POC command, the provision command, the job command, +the preflight check command, and the dashboard command. +Detailed explanations for each can be found in their respective sections, linked below. .. toctree:: :maxdepth: 1 diff --git a/examples/hello-world/step-by-step/cifar10/sag/sag.ipynb b/examples/hello-world/step-by-step/cifar10/sag/sag.ipynb index 7d9c971f12..3ef045687f 100644 --- a/examples/hello-world/step-by-step/cifar10/sag/sag.ipynb +++ b/examples/hello-world/step-by-step/cifar10/sag/sag.ipynb @@ -21,21 +21,21 @@ "\n", "## Scatter and Gather (SAG)\n", "\n", - "Scatter and Gather are part of the Message Passing Interface (MPI). [MPI](https://en.wikipedia.org/wiki/Message_Passing_Interface) is a standardized and portable message-passing standard designed to function on parallel computing architectures. MPI consists of some [collective communication routines](https://mpitutorial.com/tutorials/mpi-broadcast-and-collective-communication/), such as broadcast, scatter, and gather. [Scatter and Gather](https://mpitutorial.com/tutorials/mpi-scatter-gather-and-allgather/) are widely used for federated learning in aggregation algorithms such as Fed Average.\n", + "FLARE's Scatter and Gather workflow is similar to the Message Passing Interface (MPI)'s MPI Broadcast + MPI Gather. [MPI](https://en.wikipedia.org/wiki/Message_Passing_Interface) is a standardized and portable message-passing standard designed to function on parallel computing architectures. MPI consists of some [collective communication routines](https://mpitutorial.com/tutorials/mpi-broadcast-and-collective-communication/), such as MPI Broadcast, MPI Scatter, and MPI Gather.\n", "\n", "\"scatter\"\"gather\"\n", "\n", "\n", "\n", "## FedAvg with SAG\n", - "[FedAvg](https://nvflare.readthedocs.io/en/main/programming_guide/controllers/scatter_and_gather_workflow.html) is a workflow that leverage MPI Scatter and Gather. You can see one round of training in such workflow.\n", + "We use [SAG workflow](https://nvflare.readthedocs.io/en/main/programming_guide/controllers/scatter_and_gather_workflow.html) to implement the FedAvg algorithm. You can see one round of training in such workflow.\n", "\n", "\"FedAvg\"\n", "\n", "\n", "\"FedAvg\" \"Scatter\n", "\n", - "The Fed Avg aggregation is done on the server side, its weighted on the number of training steps on each client\n", + "The FedAvg aggregation is done on the server side, its weighted on the number of training steps on each client\n", " \n", "## Convert training code to federated learning training code\n", "\n", diff --git a/nvflare/app_common/executors/client_api_launcher_executor.py b/nvflare/app_common/executors/client_api_launcher_executor.py index 6fdc9b8372..24df1587b3 100644 --- a/nvflare/app_common/executors/client_api_launcher_executor.py +++ b/nvflare/app_common/executors/client_api_launcher_executor.py @@ -43,8 +43,8 @@ def __init__( submit_model_task_name: str = "submit_model", from_nvflare_converter_id: Optional[str] = None, to_nvflare_converter_id: Optional[str] = None, - params_exchange_format: ExchangeFormat = ExchangeFormat.NUMPY, - params_transfer_type: TransferType = TransferType.FULL, + params_exchange_format: str = ExchangeFormat.NUMPY, + params_transfer_type: str = TransferType.FULL, config_file_name: str = CLIENT_API_CONFIG, ) -> None: """Initializes the ClientAPILauncherExecutor. @@ -70,8 +70,8 @@ def __init__( This ParamsConverter will be called when model is sent from nvflare controller side to executor side. to_nvflare_converter_id (Optional[str]): Identifier used to get the ParamsConverter from NVFlare components. This ParamsConverter will be called when model is sent from nvflare executor side to controller side. - params_exchange_format (ExchangeFormat): What format to exchange the parameters. - params_transfer_type (TransferType): How to transfer the parameters. FULL means the whole model parameters are sent. + params_exchange_format (str): What format to exchange the parameters. + params_transfer_type (str): How to transfer the parameters. FULL means the whole model parameters are sent. DIFF means that only the difference is sent. config_file_name (str): The config file name to write attributes into, the client api will read in this file. """ diff --git a/nvflare/app_common/executors/task_exchanger.py b/nvflare/app_common/executors/task_exchanger.py index e7521d3380..663459317c 100644 --- a/nvflare/app_common/executors/task_exchanger.py +++ b/nvflare/app_common/executors/task_exchanger.py @@ -46,16 +46,27 @@ def __init__( """Constructor of TaskExchanger. Args: - pipe_id: component id of pipe - read_interval: how often to read from pipe - heartbeat_interval: how often to send heartbeat to peer - heartbeat_timeout: max amount of time to allow missing heartbeats before treating peer as dead - resend_interval: how often to resend a message when failing to send - max_resends: max number of resends. None means no limit - peer_read_timeout: time to wait for peer to accept sent message - task_wait_time: how long to wait for a task to complete. None means waiting forever - result_poll_interval: how often to poll task result - pipe_channel_name: the channel name for sending task requests + pipe_id (str): component id of pipe. + read_interval (float): how often to read from pipe. + Defaults to 0.1. + heartbeat_interval (float): how often to send heartbeat to peer. + Defaults to 5.0. + heartbeat_timeout (float, optional): how long to wait for a + heartbeat from the peer before treating the peer as dead, + 0 means DO NOT check for heartbeat. Defaults to 30.0. + resend_interval (float): how often to resend a message if failing to send. + None means no resend. Note that if the pipe does not support resending, + then no resend. Defaults to 2.0. + max_resends (int, optional): max number of resend. None means no limit. + Defaults to None. + peer_read_timeout (float, optional): time to wait for peer to accept sent message. + Defaults to 5.0. + task_wait_time (float, optional): how long to wait for a task to complete. + None means waiting forever. Defaults to None. + result_poll_interval (float): how often to poll task result. + Defaults to 0.5. + pipe_channel_name: the channel name for sending task requests. + Defaults to "task". """ Executor.__init__(self) check_str("pipe_id", pipe_id) diff --git a/nvflare/app_opt/lightning/api.py b/nvflare/app_opt/lightning/api.py index d34025e913..9035b2fe7c 100644 --- a/nvflare/app_opt/lightning/api.py +++ b/nvflare/app_opt/lightning/api.py @@ -38,13 +38,42 @@ def patch(trainer: pl.Trainer, restore_state: bool = True, load_state_dict_strict: bool = True): - """Patch the lightning trainer for usage with NVFlare. + """Patches the PyTorch Lightning Trainer for usage with NVFlare. Args: trainer: the PyTorch Lightning trainer. - restore_state: whether to restore optimizer and learning rate scheduler states. Defaults to `True`. - load_state_dict_strict: exposes `strict` argument of `torch.nn.Module.load_state_dict()` used load the received model. Defaults to `True`. + restore_state: whether to restore optimizer and learning rate scheduler states. + Defaults to `True`. + load_state_dict_strict: exposes `strict` argument of `torch.nn.Module.load_state_dict()` + used to load the received model. Defaults to `True`. See https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.load_state_dict for details. + + Example: + + Normal usage: + + .. code-block:: python + + trainer = Trainer(max_epochs=1) + flare.patch(trainer) + + + Advanced usage: + + If users want to pass additional information to FLARE server side via the lightning API, + they will need to set the information inside the attributes called ``__fl_meta__`` in their LightningModule. + + .. code-block:: python + + class LitNet(LightningModule): + def __init__(self): + super().__init__() + self.save_hyperparameters() + self.model = Net() + self.train_acc = Accuracy(task="multiclass", num_classes=NUM_CLASSES) + self.valid_acc = Accuracy(task="multiclass", num_classes=NUM_CLASSES) + self.__fl_meta__ = {"CUSTOM_VAR": "VALUE_OF_THE_VAR"} + """ fl_callback = FLCallback(rank=trainer.global_rank, load_state_dict_strict=load_state_dict_strict) callbacks = trainer.callbacks @@ -67,7 +96,8 @@ def __init__(self, rank: int = 0, load_state_dict_strict: bool = True): Args: rank: global rank of the PyTorch Lightning trainer. - load_state_dict_strict: exposes `strict` argument of `torch.nn.Module.load_state_dict()` used load the received model. Defaults to `True`. + load_state_dict_strict: exposes `strict` argument of `torch.nn.Module.load_state_dict()` + used to load the received model. Defaults to `True`. See https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.load_state_dict for details. """ super(FLCallback, self).__init__() diff --git a/nvflare/client/api.py b/nvflare/client/api.py index 6fd5b945aa..f9aaf886bb 100644 --- a/nvflare/client/api.py +++ b/nvflare/client/api.py @@ -14,7 +14,7 @@ import importlib import os -from typing import Any, Dict, Optional, Tuple, Union +from typing import Any, Dict, Optional, Tuple from nvflare.apis.analytix import AnalyticsDataType from nvflare.apis.utils.analytix_utils import create_analytic_dxo @@ -32,13 +32,11 @@ PROCESS_MODEL_REGISTRY = None -def _create_client_config(config: Union[str, Dict]) -> ClientConfig: +def _create_client_config(config: str) -> ClientConfig: if isinstance(config, str): client_config = from_file(config_file=config) - elif isinstance(config, dict): - client_config = ClientConfig(config=config) else: - raise ValueError("config should be either a string or dictionary.") + raise ValueError("config should be a string.") return client_config @@ -63,15 +61,24 @@ def _register_tensor_decomposer(): def init( - config: Union[str, Dict] = f"config/{CLIENT_API_CONFIG}", rank: Optional[str] = None, -): +) -> None: """Initializes NVFlare Client API environment. Args: - config (str or dict): configuration file or config dictionary. rank (str): local rank of the process. It is only useful when the training script has multiple worker processes. (for example multi GPU) + + Returns: + None + + Example: + + .. code-block:: python + + nvflare.client.init() + + """ global PROCESS_MODEL_REGISTRY # Declare PROCESS_MODEL_REGISTRY as global @@ -82,7 +89,7 @@ def init( print("Warning: called init() more than once. The subsequence calls are ignored") return - client_config = _create_client_config(config=config) + client_config = _create_client_config(config=f"config/{CLIENT_API_CONFIG}") flare_agent = None try: @@ -114,6 +121,7 @@ def init( def get_model_registry() -> ModelRegistry: + """Gets the ModelRegistry.""" if PROCESS_MODEL_REGISTRY is None: raise RuntimeError("needs to call init method first") return PROCESS_MODEL_REGISTRY @@ -124,6 +132,13 @@ def receive(timeout: Optional[float] = None) -> Optional[FLModel]: Returns: An FLModel received. + + Example: + + .. code-block:: python + + nvflare.client.receive() + """ model_registry = get_model_registry() return model_registry.get_model(timeout) @@ -135,6 +150,13 @@ def send(fl_model: FLModel, clear_registry: bool = True) -> None: Args: fl_model (FLModel): Sends a FLModel object. clear_registry (bool): To clear the registry or not. + + Example: + + .. code-block:: python + + nvflare.client.send(fl_model=FLModel(...)) + """ model_registry = get_model_registry() model_registry.submit_model(model=fl_model) @@ -143,7 +165,15 @@ def send(fl_model: FLModel, clear_registry: bool = True) -> None: def clear(): - """Clears the model registry.""" + """Clears the model registry. + + Example: + + .. code-block:: python + + nvflare.client.clear() + + """ model_registry = get_model_registry() model_registry.clear() @@ -154,29 +184,89 @@ def system_info() -> Dict: System information will be available after a valid FLModel is received. It does not retrieve information actively. + Note: + system information includes job id and site name. + Returns: A dict of system information. + + Example: + + .. code-block:: python + + sys_info = nvflare.client.system_info() + """ model_registry = get_model_registry() return model_registry.get_sys_info() def get_config() -> Dict: + """Gets the ClientConfig dictionary. + + Returns: + A dict of the configuration used in Client API. + + Example: + + .. code-block:: python + + config = nvflare.client.get_config() + + """ model_registry = get_model_registry() return model_registry.config.config def get_job_id() -> str: + """Gets job id. + + Returns: + The current job id. + + Example: + + .. code-block:: python + + job_id = nvflare.client.get_job_id() + + """ sys_info = system_info() return sys_info.get(ConfigKey.JOB_ID, "") def get_site_name() -> str: + """Gets site name. + + Returns: + The site name of this client. + + Example: + + .. code-block:: python + + site_name = nvflare.client.get_site_name() + + """ sys_info = system_info() return sys_info.get(ConfigKey.SITE_NAME, "") def is_running() -> bool: + """Returns whether the NVFlare system is up and running. + + Returns: + True, if the system is up and running. False, otherwise. + + Example: + + .. code-block:: python + + while nvflare.client.is_running(): + # receive model, perform task, send model, etc. + ... + + """ try: receive() return True @@ -185,6 +275,20 @@ def is_running() -> bool: def is_train() -> bool: + """Returns whether the current task is a training task. + + Returns: + True, if the current task is a training task. False, otherwise. + + Example: + + .. code-block:: python + + if nvflare.client.is_train(): + # perform train task on received model + ... + + """ model_registry = get_model_registry() if model_registry.rank != "0": raise RuntimeError("only rank 0 can call is_train!") @@ -192,6 +296,20 @@ def is_train() -> bool: def is_evaluate() -> bool: + """Returns whether the current task is an evaluate task. + + Returns: + True, if the current task is an evaluate task. False, otherwise. + + Example: + + .. code-block:: python + + if nvflare.client.is_evaluate(): + # perform evaluate task on received model + ... + + """ model_registry = get_model_registry() if model_registry.rank != "0": raise RuntimeError("only rank 0 can call is_evaluate!") @@ -199,17 +317,58 @@ def is_evaluate() -> bool: def is_submit_model() -> bool: + """Returns whether the current task is a submit_model task. + + Returns: + True, if the current task is a submit_model. False, otherwise. + + Example: + + .. code-block:: python + + if nvflare.client.is_submit_model(): + # perform submit_model task to obtain the best local model + ... + + """ model_registry = get_model_registry() if model_registry.rank != "0": raise RuntimeError("only rank 0 can call is_submit_model!") return model_registry.task_name == model_registry.config.get_submit_model_task() -def log(key: str, value: Any, data_type: AnalyticsDataType, **kwargs): +def log(key: str, value: Any, data_type: AnalyticsDataType, **kwargs) -> bool: + """Logs a key value pair. + + We suggest users use the high-level APIs in nvflare/client/tracking.py + + Args: + key (str): key string. + value (Any): value to log. + data_type (AnalyticsDataType): the data type of the "value". + kwargs: additional arguments to be included. + + Returns: + whether the key value pair is logged successfully + + Example: + + .. code-block:: python + + log( + key=tag, + value=scalar, + data_type=AnalyticsDataType.SCALAR, + global_step=global_step, + writer=LogWriterName.TORCH_TB, + **kwargs, + ) + + """ model_registry = get_model_registry() if model_registry.rank != "0": raise RuntimeError("only rank 0 can call log!") flare_agent = model_registry.flare_agent dxo = create_analytic_dxo(tag=key, value=value, data_type=data_type, **kwargs) - flare_agent.log(dxo) + return flare_agent.log(dxo) diff --git a/nvflare/client/config.py b/nvflare/client/config.py index 2e7b9e3e17..e85d3ab837 100644 --- a/nvflare/client/config.py +++ b/nvflare/client/config.py @@ -14,7 +14,6 @@ import json import os -from enum import Enum from typing import Dict, Optional from nvflare.fuel.utils.config_factory import ConfigFactory @@ -26,7 +25,7 @@ class ExchangeFormat: NUMPY = "numpy" -class TransferType(str, Enum): +class TransferType: FULL = "FULL" DIFF = "DIFF" @@ -34,7 +33,6 @@ class TransferType(str, Enum): class ConfigKey: EXCHANGE_FORMAT = "exchange_format" TRANSFER_TYPE = "transfer_type" - GLOBAL_EVAL = "global_eval" TRAIN_WITH_EVAL = "train_with_eval" TRAIN_TASK_NAME = "train_task_name" EVAL_TASK_NAME = "eval_task_name" @@ -50,47 +48,72 @@ class ConfigKey: class ClientConfig: - """Config class used in nvflare.client module. + """Config class used in `nvflare.client` module. + + Note: + The config has the following keys: + + .. code-block:: + + EXCHANGE_FORMAT: Format to exchange, pytorch, raw, or numpy + TRANSFER_TYPE: Either FULL or DIFF (means difference) + TRAIN_WITH_EVAL: Whether train task needs to also do evaluation + TRAIN_TASK_NAME: Name of the train task + EVAL_TASK_NAME: Name of the evaluate task + SUBMIT_MODEL_TASK_NAME: Name of the submit_model task + PIPE_CHANNEL_NAME: Channel name of the pipe + PIPE: pipe section + CLASS_NAME: Class name + ARG: Arguments + SITE_NAME: Site name + JOB_ID: Job id + TASK_EXCHANGE: TASK_EXCHANGE section + METRICS_EXCHANGE: METRICS_EXCHANGE section Example: - { - "METRICS_EXCHANGE": { - "pipe_channel_name": "metric", - "pipe": { - "CLASS_NAME": "nvflare.fuel.utils.pipe.cell_pipe.CellPipe", - "ARG": { - "mode": "ACTIVE", - "site_name": "site-1", - "token": "simulate_job", - "root_url": "tcp://0:51893", - "secure_mode": false, - "workspace_dir": "xxx" + The content of config looks like: + + .. code-block:: json + + { + "METRICS_EXCHANGE": { + "pipe_channel_name": "metric", + "pipe": { + "CLASS_NAME": "nvflare.fuel.utils.pipe.cell_pipe.CellPipe", + "ARG": { + "mode": "ACTIVE", + "site_name": "site-1", + "token": "simulate_job", + "root_url": "tcp://0:51893", + "secure_mode": false, + "workspace_dir": "xxx" + } + } + }, + "SITE_NAME": "site-1", + "JOB_ID": "simulate_job", + "TASK_EXCHANGE": { + "train_with_eval": true, + "exchange_format": "numpy", + "transfer_type": "DIFF", + "train_task_name": "train", + "eval_task_name": "evaluate", + "submit_model_task_name": "submit_model", + "pipe_channel_name": "task", + "pipe": { + "CLASS_NAME": "nvflare.fuel.utils.pipe.cell_pipe.CellPipe", + "ARG": { + "mode": "ACTIVE", + "site_name": "site-1", + "token": "simulate_job", + "root_url": "tcp://0:51893", + "secure_mode": false, + "workspace_dir": "xxx" + } + } } } - }, - "SITE_NAME": "site-1", - "JOB_ID": "simulate_job", - "TASK_EXCHANGE": { - "train_with_eval": true, - "exchange_format": "numpy", - "transfer_type": "DIFF", - "train_task_name": "train", - "eval_task_name": "evaluate", - "submit_model_task_name": "submit_model", - "pipe_channel_name": "task", - "pipe": { - "CLASS_NAME": "nvflare.fuel.utils.pipe.cell_pipe.CellPipe", - "ARG": { - "mode": "ACTIVE", - "site_name": "site-1", - "token": "simulate_job", - "root_url": "tcp://0:51893", - "secure_mode": false, - "workspace_dir": "xxx" - } - } - } - } + """ def __init__(self, config: Optional[Dict] = None): @@ -110,10 +133,10 @@ def get_pipe_args(self, section: str) -> dict: def get_pipe_class(self, section: str) -> str: return self.config[section][ConfigKey.PIPE][ConfigKey.CLASS_NAME] - def get_exchange_format(self) -> ExchangeFormat: + def get_exchange_format(self) -> str: return self.config[ConfigKey.TASK_EXCHANGE][ConfigKey.EXCHANGE_FORMAT] - def get_transfer_type(self): + def get_transfer_type(self) -> str: return self.config.get(ConfigKey.TASK_EXCHANGE, {}).get(ConfigKey.TRANSFER_TYPE, "FULL") def get_train_task(self): diff --git a/nvflare/client/decorator.py b/nvflare/client/decorator.py index 87c6762fbe..a70e0bba18 100644 --- a/nvflare/client/decorator.py +++ b/nvflare/client/decorator.py @@ -30,6 +30,23 @@ def train( _func=None, **root_kwargs, ): + """A decorator to wraps the training logic. + + Note: + FLARE will pass the model received from the server side to the first argument of the decorated method. + The return value of the decorated training method needs to be an FLModel object. + + Usage: + + .. code-block:: python + + @nvflare.client.train + def my_train(input_model=None, device="cuda:0"): + ... + return new_model + + """ + def decorator(train_fn): @functools.wraps(train_fn) def wrapper(*args, **kwargs): @@ -65,6 +82,25 @@ def evaluate( _func=None, **root_kwargs, ): + """A decorator to wraps the evaluate logic. + + Note: + FLARE will pass the model received from the server side to the first argument of the decorated method. + The return value of the decorated method needs to be a float number metric. + The decorated method needs to be run BEFORE the training method, + so the metrics will be sent along with the trained output model. + + Usage: + + .. code-block:: python + + @nvflare.client.evaluate + def my_eval(input_model, device="cuda:0"): + ... + return metrics + + """ + def decorator(eval_fn): @functools.wraps(eval_fn) def wrapper(*args, **kwargs): diff --git a/nvflare/client/flare_agent.py b/nvflare/client/flare_agent.py index a9f9c7c890..763449d824 100644 --- a/nvflare/client/flare_agent.py +++ b/nvflare/client/flare_agent.py @@ -16,7 +16,7 @@ import threading import time import traceback -from typing import Optional +from typing import Any, Optional from nvflare.apis.dxo import DXO, MetaKey, from_shareable from nvflare.apis.fl_constant import FLContextKey @@ -71,17 +71,34 @@ def __init__( max_resends=None, submit_result_timeout=30.0, metric_pipe=None, - task_channel_name=PipeChannelName.TASK, - metric_channel_name=PipeChannelName.METRIC, + task_channel_name: str = PipeChannelName.TASK, + metric_channel_name: str = PipeChannelName.METRIC, close_pipe: bool = True, close_metric_pipe: bool = True, ): - """Constructor of Flare Agent. The agent is responsible for communicating with the Flare Client Job cell (CJ) + """Constructor of Flare Agent. + + The agent is responsible for communicating with the Flare Client Job cell (CJ) to get task and to submit task result. Args: - pipe: pipe for communication - submit_result_timeout: when submitting task result, how long to wait for response from the CJ + pipe (Pipe): pipe for task communication. + read_interval (float): how often to read from the pipe. Defaults to 0.1. + heartbeat_interval (float): how often to send a heartbeat to the peer. Defaults to 5.0. + heartbeat_timeout (float): how long to wait for a heartbeat from the peer before treating the peer as dead, + 0 means DO NOT check for heartbeat. Defaults to 30.0. + resend_interval (float): how often to resend a message if failing to send. None means no resend. + Note that if the pipe does not support resending, then no resend. Defaults to 2.0. + max_resends (int, optional): max number of resend. None means no limit. Defaults to None. + submit_result_timeout (float): when submitting task result, + how long to wait for response from the CJ. Defaults to 30.0. + metric_pipe (Pipe, optional): pipe for metric communication. Defaults to None. + task_channel_name (str): channel name for task. Defaults to ``task``. + metric_channel_name (str): channel name for metric. Defaults to ``metric``. + close_pipe (bool): whether to close the task pipe when stopped. Defaults to True. + Usually for ``FilePipe`` we set to False, for ``CellPipe`` we set to True. + close_metric_pipe (bool): whether to close the metric pipe when stopped. Defaults to True. + Usually for ``FilePipe`` we set to False, for ``CellPipe`` we set to True. """ flare_decomposers.register() common_decomposers.register() @@ -119,7 +136,9 @@ def __init__( self._close_metric_pipe = close_metric_pipe def start(self): - """Start the agent. This method must be called to enable CJ/Agent communication. + """Start the agent. + + This method must be called to enable CJ/Agent communication. Returns: None @@ -141,7 +160,9 @@ def _status_cb(self, msg: Message, pipe_handler: PipeHandler, channel): pipe_handler.stop(self._close_pipe) def stop(self): - """Stop the agent. After this is called, there will be no more communications between CJ and agent. + """Stop the agent. + + After this is called, there will be no more communications between CJ and agent. Returns: None @@ -152,13 +173,17 @@ def stop(self): if self.metric_pipe_handler: self.metric_pipe_handler.stop(self._close_metric_pipe) - def shareable_to_task_data(self, shareable: Shareable): + def shareable_to_task_data(self, shareable: Shareable) -> Any: """Convert the Shareable object received from the TaskExchanger to an app-friendly format. + Subclass can override this method to convert to its own app-friendly task data. By default, we convert to DXO object. Args: shareable: the Shareable object received from the TaskExchanger. + + Returns: + task data. """ try: dxo = from_shareable(shareable) @@ -175,7 +200,7 @@ def shareable_to_task_data(self, shareable: Shareable): self.logger.error(f"failed to extract DXO from shareable object: {ex}") raise ex - def get_task(self, timeout: Optional[float] = None): + def get_task(self, timeout: Optional[float] = None) -> Optional[Task]: """Get a task from FLARE. This is a blocking call. Args: @@ -184,6 +209,7 @@ def get_task(self, timeout: Optional[float] = None): Returns: None if no task is available before timeout; or a Task object if task is available. + Raises: AgentClosed exception if the agent has been closed before timeout. CallStateError exception if the call has not been made properly. @@ -229,6 +255,7 @@ def get_task(self, timeout: Optional[float] = None): def submit_result(self, result, rc=RC.OK) -> bool: """Submit the result of the current task. + This is a blocking call. The agent will try to send the result to flare site until it is successfully sent or the task is aborted or the agent is closed. @@ -236,8 +263,11 @@ def submit_result(self, result, rc=RC.OK) -> bool: result: result to be submitted rc: return code - Returns: whether the result is submitted successfully - Raises: the CallStateError exception if the submit_result call is not made properly. + Returns: + whether the result is submitted successfully + + Raises: + the CallStateError exception if the submit_result call is not made properly. Notes: the application must only make this call after the received task is processed. The call can only be made a single time regardless whether the submission is successful. @@ -261,14 +291,15 @@ def submit_result(self, result, rc=RC.OK) -> bool: return result - def task_result_to_shareable(self, result, rc) -> Shareable: + def task_result_to_shareable(self, result: Any, rc) -> Shareable: """Convert the result object to Shareable object before sending back to the TaskExchanger. + Subclass can override this method to convert its app-friendly result type to Shareable. By default, we expect the result to be DXO object. Args: - result: the result object to be converted to Shareable. If None, an empty Shareable object will be - created with the rc only. + result: the result object to be converted to Shareable. + If None, an empty Shareable object will be created with the rc only. rc: the return code. Returns: @@ -289,7 +320,15 @@ def _do_submit_result(self, current_task: _TaskContext, result, rc): reply = Message.new_reply(topic=current_task.task_name, req_msg_id=current_task.msg_id, data=result) return self.pipe_handler.send_to_peer(reply, self.submit_result_timeout) - def log(self, record: DXO): + def log(self, record: DXO) -> bool: + """Logs a metric record. + + Args: + record (DXO): A metric record. + + Returns: + whether the metric record is submitted successfully + """ if not self.metric_pipe_handler: raise RuntimeError("metric pipe is not available") @@ -313,6 +352,24 @@ def __init__( submit_result_timeout=30.0, has_metrics=False, ): + """Constructor of Flare Agent with Cell Pipe. This is a convenient class. + + Args: + agent_id (str): unique id to guarantee the uniqueness of cell's FQCN. + site_name (str): name of the FLARE site + root_url (str): the root url of the cellnet that the pipe's cell will join + secure_mode (bool): whether connection to the root is secure (TLS) + workspace_dir (str): the directory that contains startup for joining the cellnet. Required only in secure mode + read_interval (float): how often to read from the pipe. + heartbeat_interval (float): how often to send a heartbeat to the peer. + heartbeat_timeout (float): how long to wait for a heartbeat from the peer before treating the peer as gone, + 0 means DO NOT check for heartbeat. + resend_interval (float): how often to resend a message if failing to send. None means no resend. + Note that if the pipe does not support resending, then no resend. + max_resends (int, optional): max number of resend. None means no limit. + submit_result_timeout (float): when submitting task result, how long to wait for response from the CJ. + has_metrics (bool): has metric pipe or not. + """ pipe = CellPipe( mode=Mode.ACTIVE, token=agent_id, diff --git a/nvflare/client/model_registry.py b/nvflare/client/model_registry.py index da3869bcbb..92c24eb1da 100644 --- a/nvflare/client/model_registry.py +++ b/nvflare/client/model_registry.py @@ -23,7 +23,7 @@ class ModelRegistry(TaskRegistry): - """This class is used to remember attributes that need to share for a user code. + """This class is used to remember attributes that need to be shared for a user code. For example, after "global_evaluate" we should remember the "metrics" value. And set that into the model that we want to submit after "train". @@ -39,6 +39,17 @@ def __init__(self, config: ClientConfig, rank: Optional[str] = None, flare_agent self.metrics = None def get_model(self, timeout: Optional[float] = None) -> Optional[FLModel]: + """Gets a model from FLARE client. + + This method gets the task from FLARE client, and extract the `task.data` out. + + Args: + timeout (float, optional): If specified, this call is blocked only for the specified amount of time. + If not specified, this call is blocked forever until a task has been received or agent has been closed. + + Returns: + None if flare agent is None; or an FLModel object if a task is available within timeout. + """ task = self.get_task(timeout) if task is not None and task.data is not None: if not isinstance(task.data, FLModel): @@ -47,6 +58,11 @@ def get_model(self, timeout: Optional[float] = None) -> Optional[FLModel]: return None def submit_model(self, model: FLModel) -> None: + """Submits a model to FLARE client. + + Args: + model (FLModel): Trained local model to be submitted. + """ if not self.flare_agent: return None if self.config.get_transfer_type() == "DIFF": @@ -74,5 +90,6 @@ def submit_model(self, model: FLModel) -> None: self.submit_task(model) def clear(self): + """Clears the model registry cache.""" super().clear() self.metrics = None diff --git a/nvflare/client/task_registry.py b/nvflare/client/task_registry.py index 7e4dac2dbd..88f556d63c 100644 --- a/nvflare/client/task_registry.py +++ b/nvflare/client/task_registry.py @@ -20,7 +20,7 @@ class TaskRegistry: - """This class is used to remember attributes that need to share for a user code.""" + """This class is used to remember attributes that need to be shared for a user code.""" def __init__(self, config: ClientConfig, rank: Optional[str] = None, flare_agent: Optional[FlareAgent] = None): self.flare_agent = flare_agent @@ -41,6 +41,9 @@ def _receive(self, timeout: Optional[float] = None): task = self.flare_agent.get_task(timeout) + if task is None: + raise RuntimeError(f"no received task within timeout: {timeout}") + if task.data is None: raise RuntimeError("no received task.data") @@ -48,24 +51,58 @@ def _receive(self, timeout: Optional[float] = None): self.task_name = task.task_name self.cache_loaded = True - def set_task_name(self, task_name: str): + def set_task_name(self, task_name: str) -> None: + """Sets the current task name. + + This method is only used in multiprocess scenario in the lightning API. + For non-rank 0 processes, they are not getting tasks from the FLARE side, + thus they rely on the rank 0 process to tell them the current task name + and will use this method to set it. + + Args: + task_name (str): current task name + """ self.task_name = task_name def get_task(self, timeout: Optional[float] = None) -> Optional[Task]: + """Gets the cached received task. + + Args: + timeout (float, optional): If specified, this call is blocked only for the specified amount of time. + If not specified, this call is blocked forever until a task has been received or agent has been closed. + + Returns: + None if flare agent is None; or a Task object if task is available within timeout. + """ if not self.cache_loaded: self._receive(timeout) return self.received_task def get_sys_info(self) -> Dict: + """Gets NVFlare system information. + + Returns: + A dict of system information. + """ return self.sys_info - def submit_task(self, data: Any, return_code: str = RC.OK) -> None: + def submit_task(self, data: Any, return_code: str = RC.OK) -> bool: + """Submits result of the current task. + + Args: + data: task result + return_code (str): return code of the task execution + + Returns: + whether the result is submitted successfully + """ if not self.flare_agent or not self.task_name or self.received_task is None: - return None + return False - self.flare_agent.submit_result(result=data, rc=return_code) + return self.flare_agent.submit_result(result=data, rc=return_code) - def clear(self): + def clear(self) -> None: + """Clears the cached received task.""" self.received_task = None self.cache_loaded = False diff --git a/nvflare/client/tracking.py b/nvflare/client/tracking.py index 2a8cef83a7..c04b797037 100644 --- a/nvflare/client/tracking.py +++ b/nvflare/client/tracking.py @@ -21,7 +21,12 @@ class SummaryWriter: - """Mimics Tensorboard apis.""" + """SummaryWriter mimics the usage of Tensorboard's SummaryWriter. + + Users can replace the import of Tensorboard's SummaryWriter with FLARE's SummaryWriter. + They would then use SummaryWriter the same as before. + SummaryWriter will send log records to the FLARE system. + """ def add_scalar(self, tag: str, scalar: float, global_step: Optional[int] = None, **kwargs): """Sends a scalar. @@ -61,6 +66,13 @@ def add_scalars(self, tag: str, scalars: dict, global_step: Optional[int] = None class WandBWriter: + """WandBWriter mimics the usage of weights and biases. + + Users can replace the import of wandb with FLARE's WandBWriter. + They would then use WandBWriter the same as they would use wandb. + WandBWriter will send log records to the FLARE system. + """ + def log(self, metrics: Dict[str, float], step: Optional[int] = None): """Log multiple metrics for the current run. @@ -69,7 +81,7 @@ def log(self, metrics: Dict[str, float], step: Optional[int] = None): step (int, optional): A single integer step at which to log the specified Metrics. """ log( - tag="metrics", + key="metrics", value=metrics, data_type=AnalyticsDataType.METRICS, global_step=step, @@ -78,11 +90,11 @@ def log(self, metrics: Dict[str, float], step: Optional[int] = None): class MLflowWriter: - """MLflowWriter mimics the usage of mlflow. + """MLflowWriter mimics the usage of MLflow. - Users can replace the import of mlflow with MLflowWriter. They would then use - MLflowWriter the same as they would use mlflow. MLflowWriter will send log records to - the receiver. + Users can replace the import of MLflow with FLARE's MLflowWriter. + They would then use MLflowWriter the same as they would use MLflow. + MLflowWriter will send log records to the FLARE system. """ def log_param(self, key: str, value: any) -> None: diff --git a/nvflare/fuel/utils/pipe/cell_pipe.py b/nvflare/fuel/utils/pipe/cell_pipe.py index d9a75dd51c..9a0ceb7469 100644 --- a/nvflare/fuel/utils/pipe/cell_pipe.py +++ b/nvflare/fuel/utils/pipe/cell_pipe.py @@ -157,18 +157,18 @@ def __init__( site_name: str, token: str, root_url: str = "", - secure_mode=True, + secure_mode: bool = True, workspace_dir: str = "", ): """The constructor of the CellPipe. Args: mode: passive or active mode - site_name: name of the FLARE site - token: unique id to guarantee the uniqueness of cell's FQCN. - root_url: the root url of the cellnet that the pipe's cell will join - secure_mode: whether connection to the root is secure (TLS) - workspace_dir: the directory that contains startup for joining the cellnet. Required only in secure_mode + site_name (str): name of the FLARE site + token (str): unique id to guarantee the uniqueness of cell's FQCN. + root_url (str): the root url of the cellnet that the pipe's cell will join + secure_mode (bool): whether connection to the root is secure (TLS) + workspace_dir (str): the directory that contains startup for joining the cellnet. Required only in secure_mode """ super().__init__(mode) self.logger = logging.getLogger(self.__class__.__name__) diff --git a/nvflare/fuel/utils/pipe/pipe_handler.py b/nvflare/fuel/utils/pipe/pipe_handler.py index 6efc9c67f2..27ff51ac26 100644 --- a/nvflare/fuel/utils/pipe/pipe_handler.py +++ b/nvflare/fuel/utils/pipe/pipe_handler.py @@ -64,19 +64,18 @@ def __init__( max_resends=None, default_request_timeout=5.0, ): - """ - Constructor of the PipeHandler. + """Constructor of the PipeHandler. Args: - pipe: the pipe to be monitored - read_interval: how often to read from the pipe - heartbeat_interval: how often to send a heartbeat to the peer - heartbeat_timeout: how long to wait for a heartbeat from the peer before treating the peer as gone, + pipe (Pipe): the pipe to be monitored. + read_interval (float): how often to read from the pipe. + heartbeat_interval (float): how often to send a heartbeat to the peer. + heartbeat_timeout (float): how long to wait for a heartbeat from the peer before treating the peer as gone, 0 means DO NOT check for heartbeat. - resend_interval: how often to resend a message if failing to send. None means no resend. + resend_interval (float): how often to resend a message if failing to send. None means no resend. Note that if the pipe does not support resending, then no resend. - max_resends: max number of resends. None means no limit. - default_request_timeout: default timeout for request if timeout not specified + max_resends (int, optional): max number of resends. None means no limit. + default_request_timeout (float): default timeout for request if timeout not specified. """ check_positive_number("read_interval", read_interval) check_positive_number("heartbeat_interval", heartbeat_interval) From 657d6380a3d4e8e44cdc95b7ada7c2902bd5fb37 Mon Sep 17 00:00:00 2001 From: Sean Yang Date: Tue, 23 Jan 2024 09:39:08 -0800 Subject: [PATCH 12/39] Address VDR feedback (#2297) * address vdr feedback * fix typos, rename * rewording, update diagram --- docs/_static/css/additions.css | 5 +- docs/conf.py | 3 +- docs/example_applications_algorithms.rst | 2 +- docs/fl_introduction.rst | 64 ++++++++++++++++++ docs/index.rst | 18 ++++- .../cross_site_model_evaluation.rst | 8 +-- .../execution_api_type/model_learner.rst | 3 +- .../3rd_party_integration_diagram.png | Bin 120806 -> 126423 bytes docs/resources/fl_diagram.png | Bin 0 -> 188729 bytes docs/resources/nvidia_logo.png | Bin 0 -> 4255 bytes docs/user_guide/nvflare_cli/job_cli.rst | 37 ++++++++++ examples/README.md | 2 +- examples/hello-world/step-by-step/README.md | 30 ++++---- job_templates/readme.md | 37 +++++++++- 14 files changed, 181 insertions(+), 28 deletions(-) create mode 100644 docs/fl_introduction.rst create mode 100644 docs/resources/fl_diagram.png create mode 100644 docs/resources/nvidia_logo.png diff --git a/docs/_static/css/additions.css b/docs/_static/css/additions.css index 999ff74614..a8490da9b4 100644 --- a/docs/_static/css/additions.css +++ b/docs/_static/css/additions.css @@ -1,3 +1,6 @@ .wy-menu-vertical li.toctree-l4.current li.toctree-l5>a{display:block;background:#b1b1b1;padding:.4045em 7.3em} .wy-menu-vertical li.toctree-l5.current li.toctree-l6>a{display:block;background:#a9a9a9;padding:.4045em 8.8em} -.wy-menu-vertical li.toctree-l5{font-size: .9em;} \ No newline at end of file +.wy-menu-vertical li.toctree-l5{font-size: .9em;} +.wy-menu > .caption > span.caption-text { + color: #76b900; + } \ No newline at end of file diff --git a/docs/conf.py b/docs/conf.py index fa388e92eb..57a8f9e1c7 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -44,7 +44,7 @@ def resolve_xref(self, env, fromdocname, builder, typ, target, node, contnode): # -- Project information ----------------------------------------------------- project = "NVIDIA FLARE" -copyright = "2023, NVIDIA" +copyright = "2024, NVIDIA" author = "NVIDIA" # The full version, including alpha/beta/rc tags @@ -114,6 +114,7 @@ def resolve_xref(self, env, fromdocname, builder, typ, target, node, contnode): html_scaled_image_link = False html_show_sourcelink = True html_favicon = "favicon.ico" +html_logo = "resources/nvidia_logo.png" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, diff --git a/docs/example_applications_algorithms.rst b/docs/example_applications_algorithms.rst index 820e9dc7f9..3b07038e97 100644 --- a/docs/example_applications_algorithms.rst +++ b/docs/example_applications_algorithms.rst @@ -25,7 +25,7 @@ Can be run from the :github_nvflare_link:`hello_world notebook ` - Example using the Scatter And Gather (SAG) workflow with a Numpy trainer - * :ref:`Hello Cross-Site Validation ` - Example using the Cross Site Model Eval workflow with a Numpy trainer + * :ref:`Hello Cross-Site Validation ` - Example using the Cross Site Model Eval workflow with a Numpy trainer, also demonstrates running cross site validation using the previous training results. * :github_nvflare_link:`Hello Cyclic Weight Transfer (GitHub) ` - Example using the CyclicController workflow to implement `Cyclic Weight Transfer `_ with TensorFlow as the deep learning training framework * :github_nvflare_link:`Swarm Learning ` - Example using Swarm Learning and Client-Controlled Cross-site Evaluation workflows. * :github_nvflare_link:`Client-Controlled Cyclic Weight Transfer ` - Example using Client-Controlled Cyclic workflow using Client API. diff --git a/docs/fl_introduction.rst b/docs/fl_introduction.rst new file mode 100644 index 0000000000..04cb9a9cd5 --- /dev/null +++ b/docs/fl_introduction.rst @@ -0,0 +1,64 @@ +.. _fl_introduction: + +########################### +What is Federated Learning? +########################### + +Federated Learning is a distributed learning paradigm where training occurs across multiple clients, each with their own local datasets. +This enables the creation of common robust models without sharing sensitive local data, helping solve issues of data privacy and security. + +How does Federated Learning Work? +================================= +The federated learning (FL) server orchestrates the collaboration of multiple clients by first sending an initial model to the FL clients. +The clients perform training on their local datasets, then send the model updates back to the FL server for aggregation to form a global model. +This process forms a single round of federated learning and after a number of rounds, a robust global model can be developed. + +.. image:: resources/fl_diagram.png + :height: 500px + :align: center + +FL Terms and Definitions +======================== + +- FL server: manages job lifecycle, orchestrates workflow, assigns tasks to clients, performs aggregation +- FL client: executes tasks, performs local computation/learning with local dataset, submits result back to FL server +- FL algorithms: FedAvg, FedOpt, FedProx etc. implemented as workflows + +.. note:: + + Here we describe the centralized version of FL, where the FL server has the role of the aggregrator node. However in a decentralized version such as + swarm learning, FL clients can serve as the aggregator node instead. + +- Types of FL + + - horizontal FL: clients hold different data samples over the same features + - vertical FL: clients hold different features over an overlapping set of data samples + - swarm learning: a decentralized subset of FL where orchestration and aggregation is performed by the clients + +Main Benefits +============= + +Enhanced Data Privacy and Security +---------------------------------- +Federated learning facilitates data privacy and data locality by ensuring that the data remains at each site. +Additionally, privacy preserving techniques such as homomorphic encryption and differential privacy filters can also be leveraged to further protect the transferred data. + +Improved Accuracy and Diversity +------------------------------- +By training with a variety of data sources across different clients, a robust and generalizable global model can be developed to better represent heterogeneous datasets. + +Scalability and Network Efficiency +---------------------------------- +With the ability to perform training at the edge, federated learning can be highly scalable across the globe. +Additionally only needing to transfer the model weights rather than entire datasets enables efficient use of network resources. + +Applications +============ +An important application of federated learning is in the healthcare sector, where data privacy regulations and patient record confidentiality make training models challenging. +Federated learning can help break down these healthcare data silos to allow hospitals and medical institutions to collaborate and pool their medical knowledge without the need to share their data. +Some common use cases involve classification and detection tasks, drug discovery with federated protein LLMs, and federated analytics on medical devices. + +Furthermore there are many other areas and industries such as financial fraud detection, autonomous vehicles, HPC, mobile applications, etc. +where the ability to use distributed data silos while maintaining data privacy is essential for the development of better models. + +Read on to learn how FLARE is built as a flexible federated computing framework to enable federated learning from research to production. \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index d012124f12..3e73c525fd 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -5,15 +5,29 @@ NVIDIA FLARE .. toctree:: :maxdepth: -1 :hidden: + :caption: Introduction + fl_introduction flare_overview whats_new getting_started + +.. toctree:: + :maxdepth: -1 + :hidden: + :caption: Guides + example_applications_algorithms real_world_fl user_guide programming_guide best_practices + +.. toctree:: + :maxdepth: -1 + :hidden: + :caption: Miscellaneous + faq publications_and_talks contributing @@ -39,8 +53,8 @@ Learn more in the :ref:`FLARE Overview `, :ref:`What's New ` covers installation and walks through an example application using the FL Simulator. +For first-time users and FL researchers, FLARE provides the :ref:`FL Simulator ` that allows you to build, test, and deploy applications locally. +The :ref:`Getting Started ` guide covers installation and walks through an example application using the FL Simulator. When you are ready to for a secure, distributed deployment, the :ref:`Real World Federated Learning ` section covers the tools and process required to deploy and operate a secure, real-world FLARE project. diff --git a/docs/programming_guide/controllers/cross_site_model_evaluation.rst b/docs/programming_guide/controllers/cross_site_model_evaluation.rst index 456e8fc138..75936806d5 100644 --- a/docs/programming_guide/controllers/cross_site_model_evaluation.rst +++ b/docs/programming_guide/controllers/cross_site_model_evaluation.rst @@ -23,7 +23,7 @@ example that implements the :class:`cross site model evaluation workflow` to write the results to a JSON file on the server. -Example with Cross Site Model Evaluation / Federated Evaluation Workflow -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -See the :github_nvflare_link:`Hello Numpy Cross-Site Validation ` for an example application with -the cross site model evaluation / federated evaluation workflow. +Examples with Cross Site Model Evaluation / Federated Evaluation Workflow +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +See :github_nvflare_link:`Hello Numpy Cross-Site Validation ` and +:github_nvflare_link:`Step-by-step Cross-site Evaluation ` for examples using server-controlled cross-site evaluation workflows. diff --git a/docs/programming_guide/execution_api_type/model_learner.rst b/docs/programming_guide/execution_api_type/model_learner.rst index 6a80fec437..292d0e78c3 100644 --- a/docs/programming_guide/execution_api_type/model_learner.rst +++ b/docs/programming_guide/execution_api_type/model_learner.rst @@ -197,5 +197,6 @@ More Resources ============== In addition to the :github_nvflare_link:`ModelLearner ` and :github_nvflare_link:`FLModel ` APIs, also take a look at some examples using the ModelLearner: + - :github_nvflare_link:`Step-by-step ModelLearner ` -- :github_nvflare_link:`CIFAR10 ModelLearner ` +- :github_nvflare_link:`CIFAR10 ModelLearner ` diff --git a/docs/resources/3rd_party_integration_diagram.png b/docs/resources/3rd_party_integration_diagram.png index 43de0dbaa7c33052512a23cfea1a7c56cc693805..5f9983296840ff5db8517f0f5661d94667e350da 100644 GIT binary patch literal 126423 zcmeFZ2S8NGwl1uQ3P@@)2uMznx|=LH8c0gcG&$!Cg5;b81tf!tk`W{b3X)NxNCp8> zKna2hd*1y2y-ZKrySjGOUbWWxR;uc7Wkoq0EDEe+$ByA#x+s0^ z*s)Xe$Bv<0!8iq4f>qz6gMW@YUz3wOR@_bb;n*>rPcAZ=F18+KmNurx7~xW=PmC~b z3wvi5Mz}O142E!U;54yBm^&eCojL7HT|g7~-qzm4(#+D-1l0$|4ddb9hH}8*s@&X+ za0wnbc!P6u3JM5nqxvJvP3;Z_RIvB5w6Q@j!en_lxxr8@3J7CMI~RLr3r4skxV~iP zVrm2a1%Vg8!l122frDepc{N!ok7DRKwKhiY0g&Szb6lC!7y7vs}5Tsvyq@ zlLDV@EUit!KXRtV*7nF-q%555Z9$7H6voZD{~u^nL6{+&ERPIVUD6o(Am{O z6UKW{OOWG=Ib6w^2hMXqy_>0%v!%V=0r4ESva%mc*ffB)VeWIFv&lmDCHloZ@-jJ#y8 zm}}SzNDA0pwsA3#<2Ycvvy129;+mR(wLZACcXF|?H@CM#*vS0RD(PhJYG<;q5(*lvOkG?&4+IC{>S7O?EL?1D4!%9)@*i{^km_vj>STPt7bj1^M<-Jogo~xy(NSRBhy(dG|IrVr z75@s4L7xK=`Nz+K1l?aG>9=%3!Nkb9|Fisac5$+|{-JvCeS}6ng#+Thj-$N)T{w!O z(+y$cdLVR+Fg_a~>yk#`gE{iUVS|}Hu%1VG&*$ih#3)JR7aY#}u#Ge^fH463k+&Uu ze>jkesS_|-pog-Py|JmYGx)@!q6{qK!Gz$epJxQ(i<0+wlfY{e?cVL@9+YwX%?uLz&y^X!ozQ)4&xDkR-{sU@`w43e!fp(M>9{C0)&CiGQ zGytW4A&r1q7t;ef_IQ0p@#<2{ST`I6F9(Zi=!5a|Df^jgCt>hcAY6xX$&a*6Nah|DOWp%O{Yfn0e><@}Fy#LcvHSste-cZT--+d+ zIQ|bP!nl8cA`*c2W$jS-pxlD|oCmV@ll&d@`$_(I4m$pQV38k1&41Rv0^sQ6sd?Zm z?_X$v3vRHCaLN5kY2P5K)N6D$a9E&hw{h||3$`7G#}iEM9F{d#~;qO@6G=R7XI;*!a)M!-|YJzXwYx4 z@z66ojNA_$Kj8BmeguU6A=G~~9{Kc>uZgtNfA0T_qY35v3rAB`O8E#{g0cSOKmP1i zBH8fAY%rv_$s>$F8sjI25n*F#j!d^011$h$)d8I)MO3jZ3Fu#qr4&dM*v2idWxc871|33ecn-j*veITlb zvU?Ey{>w4xKbA<86d%g+AG&i0zV>Ej&WD-3|F7u8!GirpCH^FIhdS~j^KmHpzXhp9Kgl>*TMDA5c|VD0T}i@T< z;^4n}O8kdew`df0Zjkp>i6HkmK>C1e&fUZt1b&v;4l1eXOMv9fO1;d8=*vF zKd%h*=O23CAY+Q;{LkrDJ9|5%V|_Gdd(d<=qYLuT2Sp$l53)D%`oJb4?bBcS>wog> zQKEU&JwLFx|5h=A@%`d(^BuzTQ87Bqa2yq*!+w7sF+wI`kYUtMV)RQ~WoZjw0`TF_ z;Sj>v0ads)v-Ch#`i|!5|HPahMO0*HWrA?F_)!}?m_XXn)*MKgjU{lxWf7nrU}|Cj z%K0ED10@GpXOP-OJ_k%z8sP$p1IS>`0F{d~xP~ycHc+*{#_8+^>hHga(*6s?017|k z=AmmU$P4>T3Jc21&na*qFh^wpj!40M^FyRdkA1Z#2bE~MMqKY(4#vl z*`UDnx0CDxYk9r6J-8q{lDM9upd03A}?N^ACVR+@)87_Y5tEwFNBi| z@Jsi1qZr#DoSiL=!8pz!J?SC~Dx)AmLr(PXjL;6S>z^lm576U3nDs?M=s({v1N!`1 zcT>Ot1Rzo1C!2|aibH_=TV+(al`VmPegJ|$gV_&&Lp}ImzV%m(``aW|fq%Ha^#75} zDvu!SK%@^K@PP0ClUPOWqx)^*`!Flb1}0!y3c?F!=|Q4}H)hn*3vb@86us2MNm~Oh&2p5$E-{7>dm8 zn^-!5h#a-Q9F+5s`wm{^AtIVVUNS120F5JNw@r9=L}`Oc2LFmwb%a8P|}ci8s``o4iEuCGYv<60NK9$&iyL zZYDAanh&SVY&?CsN9~oGl0Gt#GLn%}wOrm~YMx)9xRd{6OvHXb@akjmrwlm`!T-lE z|6`D_n`980M^@_S(&GQ{)j&f7l{_Qy`#zMHr}8p=8!IgP-O84;0&yE3T+OAldfZ8X zn%-Xz^I}wQ{$D&yD8Xir-)^Aya*a9Z!sbY|QwDe zMv=*d?gEp%BK=oSC~g!a#Vl(Uegh-y1_sZa8_i*|^!+D)GcK46pAwCLKA=hKDd-Ab z=oz4Q16&>N_q}@=K<_(23Z$s91>$ZrW1LoukUyaQR~8V&k!8E0D8lDNlJHXAit@K3 zBjm)5q=ywAo?<}K54i%aniZd(;{4reOu2xENE4NMq_n>P4I1Fmhw+}*ZwIEsH$f7J ztvi8wWb`ON;Eimxrr-DO6-E-+rlb+MzZ?RgfWW!VVD{e&n7BI!6i4V2E%VxaW*y?J6m@7dM<0aWj9S{i7hpfl~2-z!BHHBv0` zQGKT+0b8m&@3zwZjxG3ZNDfaVT8SfjBZO9Q>c4?c~@ix(|&+6Y@w$0VFZ^N z*YcTvH!uyF1QQ^sy<{MI5rXG&{J0T=+^+jhg3|_kI8b4&Sjm zL{ZZkfXkIzi62o?`0McEao}qVn1d^70r0tj6rzQwl9f-8yOoxNC`u$htU~Um^sNiR#`7zx+rCJjR{6j9)D}}lvamre0*AT z=R5WG%a+DIgKUqfZoXA%bff0shG3YZl|1{-HdtctX$^nt>q4lCk_a*QtKQq)((Abu zLv`Wp=a*a?HB{a$dpn=}G#{CSFU+_tjZVKfUt&}G)cey@(p$G~Q7^0rCf&Xr!m0OU z3ixu8p}16T54YZUe?m+*21h^Idi0W0Do-)p=h^I-eSJ$L*E5SK`@S5RR@d|k?yXr8 zfezEbq()!n`@&*k)@wx6Y?8%|o5O&2cY0u8AX?U|PllA-+HiYwwb#i(BW*To=@vS~Env>EQ7G$%*RGneV$h8z=47+0PX=c`<>3kkUcu$kOBc027Qb};R^XsG7Qm(-WzVik(cI#CAzyg)5>Ek zpuaKUZcsAxZfzYl&aB z?O3J%JR6HGpKO%HA_eU9%36<@aeEwX$gt_{+T|x_TuQ4{wW{o>?5d|v`D{*y)+USh zMON~f8h-BCBWB}K->(B2r)4Uv!?*mMTLmA9omg`IJf9o; z_I|3J)kjs{ttX%A_o$s`JE5b~ABwF@o>q?7<^!JjhakfUtfOA11(E;2*lnY)m42m0 z3zF;;Ja`3FT+i>1yRxvI7uR;8ARv5;fA%!ZeM)HA_6+AJKK0wTM%eMRs#qr@U%arA z%6jnNfrKmZjgRQ6YO>jdb!fqi!50}-8zw)( ziiVJDz%*)R&h!a@8n-Ssu7AkOY*o|Hdmmp@(k*X&Vx8{CQ6cgES*Dk_1ZO~F#0P=oL>cOg-si3zHYFFi$_9cZ)99w z*V^8i+{4{`guTAQ?YHZ*vb*tkg=l2B%=miI@{pGgwKH|6k%;5MP|WLO;A2GsUzp?6 zQD3eKy^`3j@l~`RqOVm;vve7+aQF$vS+_zcdT8QZ^ycOv8n#MGOiXC2l8{-Oh;mZstO$}ZHsEcHYVJS8Aa zEV8~CV`1lIuO&9GDVI*A4d&*&g!* zNHUn#*3&-EQR)oEHiwDFsio3Qfj|PI2>0mPz9X`2Y5eME)P2WbW#|sM<}*)Pw*2#P z@pmFNgD9k1Zqc){l7xqs$!;>-3c=)NSJl!YzI>%XStlhd;xo%TQes+BChFS9=WaZo zkToo^J8dXF_cT78;z_nt1+H{MIQpWI^wq17YuD=LZmcGeNg%S+3e>6l*!Am%b7v;m zWIYHt)r+)Eo74DbO3HK3`EDK$rF07G%hQWmNh)7kZ97*&s%ZN0^+QhDMoM<69qy!H ztr*eo7Cu1fWT~-d-*I1<eQ!HeP@}8oG4(az1He$eNTKQ(I4?4>^&tc9q@JGYhQ( z2IK*uF>kw4`OkL0621PNhRICBG5x~*qHn^TK@;Q8_u9_+W$9Geg}^7Z1!n7azn>{= zN`I@+Mc7LQZJcNQI+f@cv~|uJ^`r zD{op)D^Z9bd`RFVS4R7^?jFE;GGUL>f`_|1TSG77CC_$A&hgFNl%#Iq*|nQ2%S=i# z?}Rv~VN(eab#!#dat6+ZSQ>Gnhon3rfo0>H;OI4TO8~){=UfO`NPBy7iGS$Hg{uQe zt9&kEnk92-PTT87MN-}r1UMd-n~AH-`HY zbo3?fk{vaL^WKAE3<0F%F!#jRpaNE8`U50fG%GztXA5^v)^qGpoEhqpl(prfM<;`xeJyjrj8GlR^fOT_svW|Xz4qsO<=?lPMji6=cR^DCh*(}+FMTf37b zo|?TbLsN)ZQ9$`|;O|tH}`)%%Q z{i(}Qr57xQ3bkhVn9->UT+kkID>d2I?}ThE*Ot6VKIbuYeB5_y=_^_9?vR1s#MhBJ z&o%7Ob5uz^!fV*c$s>B(o*CUVk$kqJHv|Qf8;kbX}O0M3n z-`lZ&O33@^*%oKF&;onX_3@FNl_tEiHPB8;=-ZmPj3$U2C89YaZ(^{qrhX>o!gvG~ z&O{;yqwKinTzZy6^EhNsGE`J5ex!vo0>5zUU4qgwj8fA9zuiL&B4Ec(n2k0cvW4u!Hl+5Wyz(#p!c)j=XU>5;$u z>IzrU&^f<3OE}3JdLPlXpyU*y^h}k+oQuu7o1MErvGr_M;N54H5*~z!3m{V^G3EKt zBcXG2!_DZmm95X8xUYZE(5$xO1I8}?_U#)FstNAiZ6?p7R@DwUrldqfs)2T^gN21% z&eW9nlnZ%8{PmdUzO_^_&!{$R$eUglU<9%F-zK3aW10g=J*U20{k7E_T=0>Sfr(c770F?8^;pyY-T4G?dCMx$I?T?b6xH#V=K;fc1a2 zaJKPLO5@}4|z#*QMgbv-VTD^wBo&I)t^xY0O zqA|$J+Vw>utH8$xxw->O8zM3_@NR)ziL?Fv{RQ~T%MZP3^ zQe|2WCYVKgn*9ZoFcA!=)#KK4?cj>Bf5&jv*ozn*CcB%*NF$Vu{_x?dtfVEeIe}s0 zVona$>%^gOLN&DkvRBsya6kCL);~F;kD6EGotf(FYiEOCh#s#cLo6@5nPWDYI`DHw ziZ$}|byyNpcXG=xr@Ig^DP=6eo?~A!s_IB3d!9bmpQpr+e!3r`C`T$Km69#KXoTK` z%bO^Dr!i!pW7)EGQINjy>pc5?XFuTJ%j){UgW@0KO3`Rh6^#!}JiG3l`(nJtd4VyR zPbrqWFq${%WSi6H4(32aNcGsTcJ<56h7)+rEu;oEK~6>jMCHod({{GYMrQ3ayLJ=z zI=x)5N2yZo&)l3GYr!bD6`BzdeEcD978Bcg+6VZlT>_3%)|Qsg0OuF@{ObaH_vzDT zroz#7?KqZ`op1$9qFIH%iLd%kyL(tQd@QVF@~^xww{$((2tEC!3v6tzB|+dl7c?h7 z{{{ZbLhH9hRc~O+V#RI1D%EEw2&=VtzsXn&T!?0ttBX$RC>zm`+Bi`^v(~}BPD#0L zPAbD}&BEor)=srENGN4Sg(XLH!t+t^M%D$#fS!)~!^UFqt#goIgwwq&1c%SM0}hOc zRLM^3(KttL<|LVl{2K0Idks6nj{VTvrW+S+xJoV)?K{rRv8 z6$EWdIFU^chj?nmI@!pE1*Pb`%i_TGyZxQFMCf|oB7G03Q(97Cn^IAwb7~$-kB3%D z{C3W`2*4Pkv{;T+k1g4YaUhn3_)DDUzRVUIhqyr_^WKeQ2ylPkX%hW*ZJd$V&7I2K z%i#Lgfv;)Ni)~hO09QYZrgZiS*0r*CPK&qQyRPvWgqo+9)6BGL-HPJ#p0i^kn&SA` zaa!8VJ+gsUU0s}eBEGi%gJH#|HDCZXcItQz4vUM^aY`ohvNpBXGbm3i%%9~}PJmXu;H1i(jil>CeiZG*7 zi~~0wmIOn;PlWRmMb0bp1q--$d%lq>l!G*P>h%?9lz8c-GMlLvY9b1yLRQLd-w%W6 zUcihQS6{#Z?+N0XxrZ^_yVKCz*V%&@Z4vdf zU)5^vKQ*|GKkMV2%Ab*(ePf`vgT7LtEF!PS<5RNpu&TGm3;#T-PD#G|aSd~=Ddrg; zzIU@x@+86=ebJ(Ow0&E-wbBBc@WjRkm}Hi?o^3X6PH@Vu)R<$r5;Yb|vJGL%C})ru zE#DN`826+Y?EgSvzy0+o#xHsD{i+KddbP?*UQ6cmOHcc$s(WH|tWauJ%f2wBd}%AI zwCZ41|D2r5s)?=H)upHInGz4z_s|Ns^{jE{Ph#RQ5#gv%QryN^l+0pm!Pr{%b3R=z za_ZR*PoQ8}$W64`R4bv$))*?CPX*!;5Mb!10`UTJBKhU$qaQKNnO3t|V=!7*lUrMy zpQX=IiD}`KjXT>yHG9r5nS~^z3V}%Th9@dXb!*JpOp$dO;W)PREXMdnkd8mJC8aO) z6e4@o;`KeK+f*o}OVXU7g{<7cXI=UQ&cqAAZDkB@XCmcfs8V^aen!~0#qyq&rK-t* z+RZLW|AsqwH&$1QF5M9ccg!I3AfRXAaX9^q)n+(-O_!@v#Hh>md9ThOx0E_0a!17@ ztKC#@QUOBmn6 zCCGY9+|~a33I$1bGn~5wH}0)e5XvZ8bD}E^^*!v7-ndWGN8}B#v4vUmtS43=^Aw9y za&Y*&E|Wo=aLAy0xLPvLSlJxs^!C-^vPg)q3}SYobn-mWMxh{QGIN{IkeMivL2^M1 zMi3UrJY~aF9;y6M+nRTISo9q44AXSKAmM7;tA`fpx1u-SVcoqax9~*@y%;~^4h!1@ z3C~Nym2c^aHeN$(~aUxIFV@!^V*^*E7a) zmxn(*pCIN;nPHUC@88x-ZziczyB5=}zMD*kW}DUfT<{jxQ?Q!?p`y7 zdq+f(^=nFIDP|}`++&{Rf$p}k(Oity{^LX&)m!6Ul3}%Nq+I$oVG;l?aCbmxl&!xu zKGAK`XjZm-lV7ZSjsqiz?D^aYrCZ7#+-HWU+uFW%m?w)gCu#L0kWJgw)wQOQNyuVn znvi6u8eKeJ;L!H1g2{EA&wO@+=>4#0X7KhZgORrLL!OolQUf1|{P5DlIQ~rHw6Xd5 z^u6n^`H%ZesCKldK7leu5#kY8fK3FyBTN~dy!tSE0wMMlKQZ-*G9ml78 z@C62qT)$kiC~a+>>(&=y9tm}wD-?0LoJF9?U*Q`O5rOR(P}{v!Ln+_JT2dAIZMp+9 zgN=Kpq8wLAS+j|&#@S`eHe2dWhkn<}1;DyN#jdSP!hr=brDV~wjR=`0!r)e@2kV=c(J-X~Tj7{sNu&(iQb)U%Zci*9Z?nRf+y1qgYadUkxc7ij4;GKEI ziHzp3?d@$Zx`$jdGk5MzSpkU2i(K?{$ngDc*?S7fnc#|#K|)A<6AgT1fQ}}ebvFki z_?{;+pXBIy{PK&)hMpH}SCsX&A|mpgX7cjy6Q~WSJ$P6oYio;RwYS$d%d#siLodX3 zstj5d79AtyV-v8NfS(l9Ty4V8mPaQ@OU#P-0hwdczIJ~#B_cwc82@lDz`iYSP(+Iy z>}d^!XSQvuG1CgN9uK#?b+IQb{#drm+gsx4+*bo1HEJd%-kvFyFn0JTs*&Y4m#tr- zb`pZ&ipD`JtE!=K_8UZ)5w_l(7~>H_3X7~lA1pK%<1oa-Bw~REWJf1&-U{|4SnP=C zf)1y9*L4dMi;H3#8oSuy)EX17U zFMiIbG)I@-wo61$Ce-c~(K;qFbd8zbhnT)K_MCn5EB-T5J(_bPqM}02K8&xcRyU1% zMTF&X7H(iF5v92fY8Aiins^V?QxdDqki%^{$od6aNLld?k%~%lSZnK+*le=#M@DV4 zWZI0kBOox3#+TZz(gI9hZe3XNEqxO?D9opuJGgyvFwPpMNF#(MRlqT)0ILbCcHL8M zu}Bg7hyhFcp6uxY5p28`N=~0s9zt413U$kuq6{~yl9S1jMtr(qM5yYFGsDc zOZRWYD}oa4-I8O`VmTOz>O{uQyFxY{2C+ro&XzIyQSr?dHC6`T3}3A{>B)FOS@TZp z+edWG(rq-NoYbpjj$z@^67nuYIk~y`#Cm6>cP8=3G9QyX-uClVf9aRm=t8h8z{_Pt7}u`k{8 zaU)bquqRozvT_wK=cCri_2P&!39dTlJ&yooR$TwLpCx^S*BIsR@Iq=@bmDK_y7BG{ z8w11Qb|8zBg~i=n8fAU`=$iqW?`Ch)pTEb^id~Dr{K=X18o{gWzPeHT<82TNOyE@2 z)fCi}1y5_TP$##LLIW<`C_?0Upyg&|H4MmRHDlBgPO18f%X2l448kUel$Yf8)NYsZm5tmOA*-Yss909Q(8u}5ll;2pbeYY*0Yk1UD7l#; zLN4UKs4~7H6E)TCIcvxrxx?cxvUhsnq;`};l5-IA!_KX7J7=2aFc4`{G=^%Cy@|>5 z(@ah^21)SZG+Z2!ITi*eqE?Fnt7G8a71E)nJzJ@HkWO<$*B2@=pgg`<^ZXPwj$*3X#|H4ru7*XQ$T7;lF@M=O)K*uw zVs+0|>?+62V+paj`E0u@csD3i)dzRPtllc?P(|in&MM47Yhi4;7^WmW_62u7apHu? z=ogTM4eWgkesR%$;%w2Y(a?c3D{fGMikC2abH`wHFsf~JX1k}iM$fxB?>3Fd)8$wS zV#o@qe?S11DiNdc&h0J%lU_^#^XBN`@-YWL-V!(I1_=5#qipZ-HmXUSThmRUXD=Yg z3({g-%DK@A(>2y5<59`i#bnG5ik*b9Rxr3374eT~lwCPvQ@G9=V$&G(w=W0fmcOci^T4@+^T zFe?D3$ha4tA$rDLA=CYhXEEQrz`OByk@U#z<#hKqj@QLpKdRD)mo$e7F)TOoK)D+1 zYL{k>TH`Lj%{m{%0jdi`8hW}NhNqO(M`a5^Ra;ExgJ`0nh@>?Bib^? z<`rNDIjI&v=znusyo$-}J*QEK?IY6KwYW+9$!8PqJ-(_%8DdacUt|$t?@FI_gEl5P z_8zx)eNs~!f>1k!Ls4-BO*bC`p!86AH5keSW7Rwc#hV&P|l4R+>#_6EbNO#=4exdMsZN1J$*<#~IC2b3~miE|0*{mS%dbwdzeqia$VZ zt6_sPUZB6z5Yxz^S&bl>ElP>0AKZ-~P9@E$$U!H!v;^{iW^^S!ktndaSyQkyYJ#4Z z<$Qspr6sR2C_*~X)R*F4TJ4x0@I7|QohnA8HoEzSVpA`ht$F8Yn7#=`9oSW+jQNhl zzHOmiIO-DIK*r08fZ)~TxcGQ{)w8ctFp|+fuf4pgpBxpCb1iCQPI7zUhECkASK2qE zX-qKDb)qg_z6_%bG|N^=t^FWETzfsL#4aRyRV}KN>9(m@XD+u%JAUp<@od4n?hdgDaPVE^gujK;OcH^(JU!hBe~VD>hZN=Ilr8r#w%dym(1oYOmWHQLis&Sh9RN#~|;?UYr`Tmgz5! zuUUSbReyI?Y}0CaYksio1VKM5Q`@PW^N(54qL~`pO|&Ek25rWJj6>35uHce~+7)f_ zJfYH~LE}&`1O<~ekT6|_HDxf2CR~@1k&)F8XieOBm zUn5$Xfw2OVm+S%m8;_=eQ&RMOTHUY6^6rlu8}kOq9v=!J_cP8z>p_@=!h`0;{tG4b zZv>#z#XhSoq=uR#)@5^c-1WKii7^^VG1@}6EMv(-I_hFNJW|kywB?}zna~RS`l?S3 zv+nXCT@KoV{Tyiv@E+m*0ZQY(a}Ev1izkcn)MR6+qc0^e)jae5(kJKmlB25H7q5~G zQO{gyI~FonKqwFEs(V1s7IJdlOp5fVA+Mx|pm@cqn=Bt}nvh7%~ zVu)F;X78}1#9J^bzhrIA1Nnj zQ`72iv#B_(gGSfwmFGJ=k)Wi@6y5sagD*BXscHK?BPw}1;hrS5~r2vX_dAtdx*i92gD*kHi)a;val)qyXS-6(7wgQ~J+Mv+yGpcW3}D(kefH!D zMW>Vmv!E15pQE)a#UoCtVR@2Hw_5j4c8k~>w~gtChf}>_bi?a6`BOeipK;HDmteRL zd*5NUUw16Jnw)i`K``&w^KbT6-h;ctvmjsGcVFS`Oi*yu|D4>m=Je|Br-V&cK3d3hr~dDim8BM1lv`;jIf+Ww#39+Mm@`0 z(^I>~*#0@rZhC1~z!sF_kd@$ux6#h4i&a%dZ|+R?7TaRIDz}CZzkhivpB{}vX=-n7 zRa!!ksM3am@20MSXS3A&P#Gt7R0J|K0p!^16cZ5FZ6AQWmm=f{ zVSxxy>~m(bT~i(BGLBh2e9Pml9TE7Q*xv!U?{(Zvb;gn1U+a9{bn3THh}N5|?OYua zZf0ybgjeEjlbllzzejc9xB^d9_@gst&MT|jj!LVeGhr|{mY&RYe3u31E}WB4uu)3_ zwWgp8{=_;Y)Af_%K_dme618ru?d`=l8?@Fw2t^vP)5DMMp-XvF-ms4G!G1c*o94mO z>1g2rTvY{zhH)}%Pa~YM#sng4y~ciAwq@jQju#jV$3q?)KcMk$XLI+|MY9YYsqA0G zY)u{r#gHZdad7=?8&=61zA+(&eSce`@&^j!us2BX+#cMa+bz#a`V}!st5QVFebwo?uwJ6r!Y1LoLR+O_?raLX)c&Z7`-`OxODg z{j6H|@ zYq2W#<)H+MI%k?PIdrRgZ!ORy3#XYWakmvelcpg6+fc)Tv3u1gIub;8)6yvbzoo@zEqPxTd?hB#3A zkB({!3%>WN6WP&n`81AeIdUs3MqQYxO(7Bc^BMm;ceIJ#H6yydbC6y5d|B7;OM-9s z*b?c?>o6cIxtPN(VHZBt^G z4d6WnaUcULjhl*}LU;$=tmHfSxo#ioev<3qQ5C8TcU5Ub=w)lpNjeLqn(#U+fb>&t zodET>U>Si2du0NyndiAH+WZx)l#tusgK+)v_B@XFTzBV(y#flN$F?fya$oT_U&0OA z7jq1B^;y_^qHp!np7gx$vbmZBcJcy&iayH^5RA;y(o)vQI2`|3S9LSSq`*Veval~?+Cpy0v?5ke^iWxK*g25BtaKG;Hro-%+O{Z?12gOC=R;S`_ z>Yry;CG-sGH+AVIB9+c&IS2OEwUtSAKYzHTAA_5s*JPPY5s4u!>|Z~$>mMpwOX+zp zHGVXo^5H|Q=cKEVbdtx3h}Ci|ZB&9Qc$PMCSsZw&Kci!vIUiTWL9%mUIoeyv zlQ5;DNYYoHVt=oH1rZQm4jKI&kQo6S+DrwZmLH*-H$uX2@P(zvhc-vxv2l)8C`eez z-nw};w53zX8eiT^FY#QUtS4iaXxu6m~rg`HPZLmjqY0B2*YQD=lMiKhJ@!6u;rlezpx^p(+&*SXqS9?EvVoj3XRD3MU1a3IouZp4_9UPJf z%c9o^Rb=2c?iliztsN_UYC-?%5nFu9y_K_yX?Yh*S-GkxgKJhT(nld0RJ-`HUqsjl zLW-afH)b$ag2tE5Maf*Y;(?8+3v`Q;GoWQ(r18@cj=-{BnX;W`YS6>4e4~-HKGifdS~reGG~1jb zW#+Ey{d$IMbf#g2st^!?OTV({UNz&_pDEkc9p2s^=9=sZlS7wG75DXJC)M>#;cz1wTv%;6}(U>$X{grMe z=I#U=w9ifjakWK2eDGSL+qOwX#?}c*_V?qk6tT3r+GmD67L1G^C`JP@_wkRHx+26) z2*Kt++2z|FE?vu^d98}F*L*IUoame1S3u;c7Hyb!qI9YvvbdSq=|U06bh-u!|o>Ie!%26#i#3RS1h zBgsZ6fGbU<-*@Rn=E0{Y#%^ENTVjFmRW6e+)R0fR;ek(ZPr zMM~EOAk)GW#p^!~6zQPhUH3gHn);PV$VFa4_O_y=MM@B<|Ms}H&8TJ+VMs>gn4*Vh z-Ko63FaWgUUzS-lOQ-oOmNmnNHPgc|(41Up(Wo(1)dmc=S_QBdE5xWTtkK7#4sDf! z*gX6FK6~l@I34m8W4q=p^Emv2PcxG$eXatzk8ufSC%%?kH79=g>J{c@>GR<} z<^KCyTuZ?*J399JR|EIUwvb~0SYFaBVhC@Jya&PI1&c9S67m$R8i*TV;qe;AxJ(WI zqSSoH#kg{ZF-)$on?M^*qkUr(thRV7@gSx{PWCJ&M4N?fN*e9k*jtE7x_>ySf|RUN zC80v10#cyxRgsDmp_7i)H&p&M;1?(v8_H3> zh@4q{(bC_`nOly5oJU-m5M)03`b(Z8Px62)D!5YbC$9 zs^VE`8XnBV+v>FD&g|=pUdO0K)|z@Lg9=HGi$R7HYfsHRF%>dGkJJ^qUgSF8xvV(!Sis-)g8J0x zUiytY^O7c&D&wC>GV#f$E~0tryoKl`gshtTg>V*m3BHmgXGHU7D1GVx@AgC(+`X$o ztH4Z$7OoB4h6W7fIF|0%7M-L$f3Sc2#9%DNJ__f7O<~ioQ{wfIQv!kNx>fE^cdr3rFslTeWRK&^0?qh!LkROm!8*ZbV~prjPphvtm}OQRI`HJ?mKsN7whMbRhQ7I z_daU-@P4wbSkR4ZmVO^hFkWO(O*wS z#tjq;!%h`mjSvsQZ=bk%qx-|o%A%)%TzeR=Jd%gU0O}C`j5-kRm$JQ-P(Ci3_LO=J@9XME1=Ffqn#^lebkV&W3C zK2XU(B!u37>FnE5ow27YuZKIP1%#tn12DN0Ckw>Hg1SX=@ToYag#wZSLa1BOt*ord zeAPHO1y=Zisj+7~2YUhXyCF}Y4(yA_4{!@v6)5&uvnYJ27q*BS@r|mrzcBmpG5<7$ z)%lye$9gg@+&k%|k%T1FSwpFF+T+UP9rV@)7Bh5G9dLH8X#wd+Lh(VFYOby8x3oFsitOKJVwMHlxotSK@BQ zQQx3`eHS^MKp^n_%sT@MQTS;g3e>R6`~eA?^i-?6tnr7+5sQ0-kmGcPW*{Tq5Ch2T zHzT7Kv%VP@Cf-g^z5TeU^qvbQO#6JALAzoOHmN@oe)}XzXl2Ls)wka!`KPV)=#Z+F zEdj`|=u9XiLU})+fEeU1o8O0qzMSGBYF4yMZ7cNh;cK+b0aJ7WLs&0j0NZ(eUl-t<}E8xxQ>`g$H;t>i`CM{2T5+?h`PbC$65Kx{&Jdm~;P_!yHoJw*_wo zqgL-3fXeB!%mY|-{^s?yRGB^3aj2dQ&|R*P3%sbWCj;{z)i&(E!$K+c&PUgCGcZ0% z(ijhlc7=Z@x<8XTVq(jb26}ZmMbhTGNioJ|FgV{tysMQKyHS!6ElRb%awXMj|FC2l zGPp`Gld43?K2Ip%;B0UIV_#-&&vk2$w*?D*kIgCHzRlTr^Na#4z>0jn|<>XR85VbhM<6j82{3fPw!9@gk(p3cP_h zvc7{2p7dCVnB>k{ zD`l$8c_~dJfUduZpnSHuKwPn;OPq!~5DDZV7t==XT;$AYgRqylJF&S-iH^vH?9~J? zMR7&`E6OCKT?X{3m@_plvK6qmTvA$>xU7Ei6`VMgweh||lPj5jW1Tk9`_`HMg(NyC zX9r@igA($9zEsqlF~++P@B7!6l14^RrO3rP2JoiL!;$k@wI!>N`34bW-vFHjvo)Hr z@!}(|!aTqgoiC=qiH7<~0j$&r653paU^wO&Hz3UV}wS>5>{)c$1axn7je(r2BOj z+$S6AjaEgc!HVl)7V4`9Zz+170f7pKFmeg;Ns!TB3y;;oDQF~Cl2@?!zthz0_^r z;po$o_~X+k$DIuh%E$GXn{<|J&`u%4L>Eorg>Wf6)yexnhj_#*3$ZwLJY%``%ak|N zYgnesLMCZHZh`|#;?OGGj)i0qzW*#xyFQ-~tL$)evWyS8T0J>>q%B)^bdq{ZXcDxg zrh3?l~rE6lxb2$X|#@;)4S6NjZ^MI8VLu=vXgJr)ZlHyncwWc`>V^0_*&3;^_sQ z!7MfH3+GC%@j_O_{FQdo!jkePqSy6q8Mp7b4CG)0o%8Rq&UNgN*f9F?Mu-aHTV4PO z7dJurR9b6Aeft_Ffb_Sz%NM4~CXilm{MrTBsH))b|A2rb>!$2e^n2)7BQ-1b#aNGcFKkjt{{)J?LA~5?Aoiq(boe#sR@T;KE{>Q?x;(@ugzzGD@hLnbA)D_rOI9+K zdgPKayzDE686FrzFeK0x)CY~mFO`68O|h5~3#N~d!2TLglzj_G9#Z%Yel{VbSAa7^ z0Bfv(gdVPgdk_B(UkV#qd(T6ef??lehd!koS~Lk}hrXb|(x+a>A3d50cvB?=@Cri@ z&r>Va(XtzKMOJ(SB{mNNpa(N;H$vFJRuN7V4OpXoFz4U<$9)% zwN`Eu`!YQM^e`5r@>@OO)!YXLYa}pud8PALztk7O)_JifYgS+}N6s7xjC37A``Fb$ zH%#xTmNU;bv^9vO8w}7}{jS6Bb9$d0S<2rTxI2m*kp@jr$H?JgW?A)YyBxj6Yl?gT zsRBs$E0#YjLxK{BWWm+Q+PT+#C8O(j-9Bz)-*GPDSk-@&2dSibbP_K1M&Q;-#`4u} zRUsZTp;2Q{e!EC&Qo$i{IYN@hG9C=VlYxxQK36f#r0NXv?i%UsADqVp!Aj%t1R6r{ zyx_Gh32@YaJ0D_zF}~FylU%-f-sJJa)q77qf)NA~#g{8B76(?z#znkxq&O0_rlk#^ zKS`?fR(xvt&D@D2aFX2 zjC`oPG(&RPUNz5`VGam5$QIA78+{eM@wT|9C$BmOSCjo7EA|-F%CMVuWu8>Sxb-7m z{6yuTplk>D$pim5`nYmlEcz#z5vj4>Lho#np6`PT$l~l%B7G>%E}H_+cDAI1zb^4w zOseJ4QHrX;l+b5b2&n51@MusznnC})@ z3e3b-t$o2A{ zWB3?Zb%{5@7y6gD`2T2&sB)^#?Tk<&6U7H6C7Hkgut9hYEm_rBgz?xIqeg9^iC(QZmi&8P(y?}=xk5#XFy-UyFpDK>rm@HF`n`^7Ss5NW2zt;3!$MeA(02N(+RRu%KL!VqIHBUM8gZ`stBcL)L zM|NqP;)dN25+bqZ_Kx~$S>wLIZBYCHkY##(f9a4-y#=1^&&{*-D@&1iVWou^Q1&dt zW7A4+r29W*0H%k>q@n#S)-i((&~xPLvN&0&@&2Ft{CsVo@vmhx(XqjEU6mnYnjRgG z{ei^sl;a-(TZ^Zd!bk+_gQ}3b>E-VXW&J^2jUI3A2sG!0kw3TQMQ# zE2u;D)M&D2w5>WhFGsNeOYDJy34-1K>Sa1WT?r^U>KL||CPrI(USJ%}^nooI)fku||vm-t+ z&g-Wy(_bJJ%C(fRItbl8lv%*~#M;_{Is@riCrQo>fdh8rt&WxH?;h%C&x=g`n49`qiCpE)ky|HN>UgsjLHn&gvdWsJHE|;+#+wNcrOtyS6{dH z+zZ`!7fcx})dlP8-77;)Py5`hsX?Q;x+9uqy~2 z=Q|}yJR7JrMt*cCD8dY_C*Wg| z2;y^}#VVWJj_uB^q^dh{hB*OFhF-!_(Itm5c%W49j{mZ`mNNRbp;54Yw#{b3hcE^+ z)3ljgz2CJbir3}G_Vk?cpAOYTFxb5t?UmExNwrz|^iN0idRGE3^{3*46!@q+e9O=7 zdU9(>OSi*CdML9ej~2@aS{&=55i9qwZFOaDp`w&j34jS)Sn0xld({!R09SQTz1M$i zxj(3OxgWhBFYv$drS8dJ`?s+}&dN1yxOHBPxpcX;Lc;)FYw2Jq6{U7GQgJu@?qDQ% zH~H@q!B$D36-U7PUyB71?c1F=fGf6@NwRepV^MVESocytFt0zhJx%5bkssyQ2MZzd z#jC$haU=g_b^rRQkvqsy2=)2B(0Jfwn4(*5%^&p9NFBe;dSq<1&9wnVXKU-@C9Y?0S3pb9vTTkgp`W7v>(K z2CxR-v?3UzG=R+~>79oq7xJ;-2aa?VA+#?z{tBxD+)*8k6S}eK>7dKj z)CK*%OQHJbBGaptZ%fJV15H>gqG|=$H#~;sS@M5t2X;HQhG7s5IS*2;rZ+eTcGqi6 z^gm~?knDw8B4d?j2(-_o*&ZINt;Gu-7a8LQZU@X!N=P5cKaF6a&3gTXr(uE6NmY(V zzLB%-n}sJhDF`hrD>3W4XVQjK~rc{Bt8ACeQZEGIMW6dG+`^ zoHLl+LflY{>zkq!cD!|8u5h|RmQ}(MBK$Q`cy_wevBRVJ4cDD+d{ zpmXwPYr`t;q=YG$s&U&alW8T>p=Q=0Byl3vZb_@Xg13v7^4jj=&O~=_oXk0gAdo^F z>-ETaZo+4);}-qv4cUk+g!3sxZ~SR9vl3+Z#EKgwK=rbj!LQcJ6$E!t$Qjz8`%Y%O zk4TAkl#wfWXOE4vb=k3D^94MJ@)^G*6IJV?VMfrM<(Oyl-NtSAACp057mJ18O~~|& z**Fj)&D6T~9kOb#Z>m*ftK7A4Il92Q`JijruunZt%L*MuI>t>a_Z~M|oDFT#7lJJ% zi-lLP-oGrLy9#%3!H%0fSiTR|5kX-U@(`hHbR1%wNJrQ3W(62bfBshN3t*KR%sMa5CCH7xv?Ba%<*ByEf!aM`;fng9{b?I-9btCjp+R@kMX&za zFH_+4DCd*=LBCKU*$1I6&R^le`(_U}-NU&;Qf+@JB1XNnFUY7KY!hbB_{zle_21-# zH?8`Y9IUw0y{HZ(Xerpq@K5T$l%@+L?9;%X`y9#bw$V3Y9}hezsZW-Ycgk5ayj;94 zRc6(m)4i-(=B^x0TUm@6&Vq#o+1Z(FJQd;Fui#q^AM~v=vQ`zQlFg|%5`-%pYh4|{ z^jhBCqstJKk96l%x^=9*B%Ai!9{z%SLQ?C|TyT~5b3E%%G!k=lMw zyEk{E!7h$gny#FbQe*h`lcFT#^bO)I+6@kL?0D_#W&G(qUxD@TVjvYt&TOWCJn1N4 z=G2gzXMj1HHG2^F_ITf$yKZ+&mOItqmUS}f9-;L%9xoO!nNp^cr1@veyAzyEF+bJ^ zpSc?LcHSXY(bliEJCBu2kGVC8aN{y`nS1LP^LPxcA2e49iR)-pJr9~BEX_L#iU(Tu zKf`A-c$Dg-UGz>l!vA{sH^{KPU^3F8f6^BR>TrvG&yQuP(nydvrYsqpz?`3#ht#vk ze%ce6A?|q=P5sMuqxW_q4^Y*jt0BRBxvGV1lvG>~Pd1oLLjL!4{Ov6Nmf6obTlx%L zZurKX%*P2Rm3XjH1@buRO48RhKn(`!~S)HJUfD)~BH7tZ|9_WLZFoi;Vg zBi|C8ydiOt5I-@yXz2e;#r+-v1EkzV9J?6Li$_k^H7tn1nYZ&dT;r1~Qq)7%zblA1 zNTdRYT$WqU`_ngph|F2)wp*uanuf}e>g|eGdp_O7;;)}a*JiHwfSF6U-j>*xNxJ>)5*@mW#5Eh{yH_*csJ-Zn0;Ygo%yaPJkClq(}I@1 zq%#Ty5pw-EotML-Q7Hjh-T>>MfM&BOWMAVywNwFs9)eRyP>scNx?S7~fc zGc%BpufW>kw1-wFOr5*Ou(vxysi3oNTl<*wrEwGgvx16y3zlQv2Y!daV!3+Lba*5g zYzhK?+%Q?i%=&h8$|W2=IjdB*2_DwTBvV z*4Ne@XO9)^tt-lZ-gA2?JB#_{)7}I%c@bbzlZ%% zyoxSZSQwKNI~;_lQro}o80OgQ=;9OE90|S}y7q)dAUXoi3 zpXE$4ZlCo^rI6D4b7DW&wm7mRz2iOB+TqLzP9Cyke+p=e)Z)uP>e3x=u=>@6cE7;G zy8Bx`&6U%r^r;=PFx!w-f7)$Z9r+nI1L@6;bz>zP@~HA((8vd{%8E4iabQ*e~#MQ z8?!}jV#j#>)`*mkZ=fXbhL{&8o&D}eCAzbFVJR@F_Q~F(?P1!DA0A&9>;QL)cKT!S zSt50+{3`7=GFiJ4EP#v^#PrQ*;u~rgSpTg=)f0jy>~E#0j+3G%qh5QEt6CR6eEr&3 z)Z)O@LVSfV(IYf2G3KgMXCzvO5nx|68x!5AWJ&x8lQiyP!(O<)lnJ?F4?s=YCAn@H z;nhJ)JB^pVS zgbs3(hJA}VaJl!mhV6kZki5~O1MynW`Li#fN!J?e~f4yPVbp$ zajtVFdU}-O#9K=DU9Oam(%5L-?j1)O7;N?3rk{Hwqwc##4WeE+$UA&Nrpjnsx$@d- z@^^|pv<;$f#soJY2X;HxG}*0;NwXHuHuS1(%ZgWLa0(<{X|&!n{KB=k{WrI&kFqfx z1L{C|Er2t}ReM}R{_a9NDLc~&pA)gEuIwGg)+Y(uffmzR|C6#e7Z)t|zIN0p)TvN{ zYi9&%3=I%8qyOJ)8pQipAu}Lk{E4^9E42uDy~b;?hThV4OCzA3YkgMzb?FO|_BuIT zsl=hD`kN3E;9v}N*oW0vLofSBjN$sljaESmJb3sE+UAG8w#emd2DNC9E}ZpF2h!8% zD?CM=Jv-7Rp&yh6hUju^#@uAr7; zi61A{oejj9&+W>tKlZ89zG|pGFXOFq;}{2YEvECt9|_5xLpzPFWmjjy3nM}Njz$Lq z&uedmp9K*=I%yBI3bmeAE-5nZwivpC=%t(Y2fm3)Pf`d+nocz@dv7Z4##c3s<5qZ| z-3&%*XCvp@DUCKR;mOrK!vr;$$!jaJ=n_?TIyQEtQCC|To}egta!^X0rl=_{Zl-V_ zFwF(c(4Ywnr8l8?3u>V#dQnl)V=GpDA-A!F^R&iseBb#65#T~Vm(CZTWeFHGsi4z^u z$_@So`0LJJ&b_nEz-VZteP64qXjr>LZ?a2V<#K>50MvT`8fW4RhHOtEHItFY2c5j- zHZ5rKFiQi8`pM-_7Fkul^}*%2D6HKKY914PE{Q3r~&c3;yW7u_V%B3P2vzw zICllQ&jxbm#0+bF^hi7v)cetmih;+IKt(o*<1W&?k*JoeO9CYM=Q`N32)RgkjW{Vr4_g)i{8ixOH9@ z#v;4{>I^acU*B_Ih|q@_nqI|Z9aH4>qBV0^9Ck;D_g4rE=KOQ-w1U=_6*-+47oJJE zD!lg-tNIo8O#+nI-fciq1Nk3@A^)&hm~QJe1>u96=B5R!gIAg4tPp~b`Lg=(dE6_4 zBa9l%FBe9yS;V+*t97mAOTkT86bgsluN4Sf;*=vd8ZS4FSCfX%&;OZgvHsbye83&* zgRZeft7<>R!w+>t6`ADS{3q1}TKm1y*c_<$V24`*(nph?nyWhf(kVw>VdrR$lb7UD zEq$%i>+FEwI1Cyqay9Iy3{v=m%V+MQ5&%1ZHt2j@15_i55vBH>r|JTMhK!Lwi)ZE1 zf6=Nr6yvM@4p82$HT}Ty(8#DZw(fY4_AD;`NWRMM>qOy`TXFsPxMtraTGX#4-gnc6 z+f+xak8pI1ghj({++9um=pPS}&qZ?1*il!=P4=T{GmHub&vpP+Eq#xR>)Y}I;*|#z;=h~ z8BgDvCjzw-?IGs`SETOdphCy9>)e!zKF9NcOC(gjhh3Rg*6Q-a6ctWCOm8r1@v7-oSdWo;zNkane1?s^osBK7XELtUh);Bk6)aKXy#t5!f;z*=$-JP z)@X-*(DzDR3r7urlh&3MdacS1oF2wuhFL&{!2+u{UPn$WRQ1Uw@l86z^`pn#A%oZM z823(s|4cfU>r0!&W_Or@?L0()by*C9Q@NIa#?S6g-gZa=>}Wc8?At~waEQmYr{0mU*G2ljqgg6jr1YAsxyxVfBo6D8;>&nbwGVkG#?TrwE|J zVpAi!W`1^kOY->WGkZU{_%Gymxyf+4INdLH_N^)z4RB62IZaZeLeiqo&3o{VAx)8v zc7|CGE(FE*+M^r9oeAD*rJU_~k8KImoav=vo!n<%@oQbs*LmVTyhoC%!HfMjiYI@n8YAg19usoCB<-HPQeR>&@ek*l| zQ&y{-`G_SA&EnUJ zk}h}ML!>U5r?g-tdOmy%Gj=Ly+ZuRD`9H4^{&yJ}DsgZHFU4Igap=2iuK|#xq$B?E z!*-Ti0~zFYqiCTAk(Msf}<5{a<$6!9^XZBW(L2pGZpUZ$5K+z9fR+$z~gC#XI z{oe`iYeg@=GyK)05Sm2BSlV=&Oz6+cHq=R6v#p?Gj)3fJ)AFIx$vILrVwFiZXn;+w zg}6#6BG50^(RiO&mt+^yiU$4rhv^Ta<)uvQ2EVzDR$PdIjk$GfIrpZ3k}^z$TiYI4 z?45D!>mhIpK>gi7*veJRxwQhZ9G2$Uf9?!ck5jfM!aKUCDRFP2m~6C9ycwFpv3BwN z<1&cf@wPxAJxn=QAY4O-U-?ZNR%IB50!l7U7%xRR|0=N>lOIW9+qRuVDwYXaU|;$p zaBkWXPEHxqbH~TJi{YKisP(HWOm*GUj`DwTWozm67hZB}bZYNIXN{LbOz97PVrq(a z-R9~#OK}4yo=oOg?P^PM-&Ol(bwI%hejdM+U?&qypd3C&PW8$HooEO*`LVedhp8tQ>0)!eHrJ)7c=&g;&P+4-l!?N%?oo%^NkW!CRWUIkY z3{L7ly;$aLLLPp|l`W)5yP-mO!)gE##NVB!hN-GTX%kM9n5FwW zA?tp%2?Lyt;FLTEG{$;~uOknfk&5KYmxRg^&*n+Yg&otsub)t()?*-6{p)r9oY-a+ zvLV3Rx6ZVbHD5XX+sdogxTA-}***QsP>72D z*tJ{EEprmO_*N_cuTHSfsYSGrajlW@Gnhm%Bcsh)8;zn~$opWT>lmJT-<#@Ai+9yF z?V@ebZXL|X&0qzlV>Q^%;#GO192a=@q%1t?DX^%TCz@;P^Q(OL4>1` zq9OZO6AK5uY$t@prfAOY7Gk8b>9E(y1mJ?PW9)e`q31w8pZ)T=+qWPDc%^5Dvnp2ID`F$P*<0GnVwbb()-EplT3)v znM6y8Yk?E`#ajdI(JO(>V{CYZ9Q7Yfb5`gEz9oXK>sotz(zYRI`#nv|dyd^?7k{)X z#5Pwm#_WUKBY1q!1NkL8!#LM#vBwj+?azU`VT5u%$D_-m}=sLU>&x3r(aF{PcgP;Jwo&Q3l=Du{Dv_9?&l{4%?EAZS1g ztF?8fqUy>M``ML%(+t~Mn&bX+r987q(l&aZZa9%?|beE?OK0@ zNn|enD88S`lFK^{{1<@{p68t!LN%cn3dYD`xot=Q_Br+D+%fBoB-?S$8gR zp%IxQ>wZTTQvcMl{tu(-g$eAZGq3#FP<(t`#^YHeO6_qz6bi3R5g&s9q8t%I zDAau59z=mlVH&$ADMYR3ZG4tstKffMxE)2sU+))Im{_RT@_3ET`sY7?M?2xb=T5)2 z%RLBx=woj^j5IS(o;8qoY#v!hml621KVZy=`>wk&Lm|Z9U%Hng&FSMMmRJ{Krg1|s zhjH{SO&G>IWyzX5;C!-Ibws*>4Z@11cFy;j5vM_xp1R%oV~@tqs_iwc9XUxi5zOu% zgC&P;&zG1p6n}EP6(yo-X>`&^FH}f9Se)Bgr(F!cU!w3lOrF*DSpInuN!|BMN$kKm z6OcrWwBESTKcd$7IY0EIs;QOltlQdt*y3;d=B|I8wa{4&S1KC<0FC;?{3jbw6e$;% z7QWK6asE!Ee*v;H@vm^9ob3o&w*2wqoa2#Bu<=K~d~R-veQ*&bR`{Jj`IK0B;6sa* zCp;{fREQ?(sCMNFBNsQ^aP>;8O#0bNM51S=4+WEYKzsIH>3aV_Q(D-%zPK--4~Lx zCl+BE2|7CmJJpkDoi<1Qj<=pVKi3HOYP1@+$W7n#eL)tMQUo>&)bU9K`hzkH7a)!) zOxff+)`RMUsx^qQj4W@gQBK1Qvn5A4@q>K2CVDyRdb^(Ch)M_n{5tJtHNphP+AlKW z2`YV8t^EnYJHfIOSbjO5_ML^94jqFpZ*q!>WXDX3LJe{&6qYu;{5*8W-wHIG`^7ch zxi8CNw(mh{#U*g}m=uS~ATbcymMH{IFYXu~-fO4U%!omk zuk20mU7>T4==$P$w>CD|bW|&u89MUpE3V(d?JS6gpUqg;rPo9;w}y0xvoUpFPU`UQ zRmu{<3AR2g@s&15rY7C=W?!)6Om4=RNrtr4UqR4RV9afOHqF67H|ZGUs&OICkN#BS zV1PklVK>ig=@lBcD7G=V!*Q)$bE5E_BRohR7b6o^?DH?ooL;}`AFb$IbxK@4lUuni z;J@z=@WJd3S$y&oLRG5}6%)&oFLE^wf@o7t%%!cEs`$fi=gdSD{Ui6vY)Ije&tUby zkO&hUrSh8bsJnOcXui#ySR@aOLFbn+r~4EKS*l*O^@QP%IdEGLE;zWVcUx_1{xi!t zk_w*w7o^}~GlbbKS&)H^9b0gkP|H2@JDsvS`Ci(4ue@QCfYNJb= z6-f=*rCBt4!{t^;pzY2MTa`HB*Fg#oh+VpMr~KchX}!$e)r>Vesn-qHw7`K>C?VcD zRZs~c)%se5m8>{nn%GhZI_Jlzgris?yw1RS#~rf^uVgwLEbDUDl^pY3XI1-+ty5Q9 zTNFMil)k}PuBWkJqQg8CD5_RveGK4kUL(f5LX7dzKv0TMlB4{w-{(Ek8i6(=9J(22 zZoy@0`;+YTH^kTD9aG|sZ%CiI7*%jw!_YIZ0)lqOKG>o+Qm4N?%XCwA`6h6Sw~M>$ zD7habN<0$B6aRHbc27tPo-p#3D$FD-9{p|%;gC)}_)(H@f0M!5BuyroGUFUuY-nbq zBu^90&ZHG$M~>@1e(?;BFRD^qA2kGXHo|*^rn=(l-0payLhrM7Rmf| zmw6o`nx{!m(2~;kh}eI->1>$4S<|cPOa!m7ACO{4XvY}U5B|gC>r@}d-CQY-iYt$0}SRE3rrXW0jiKTH4G*xOaew0 zDEp7kz;HUse92k+`ahYI(lppSJ3|@w8gUWqe%!iG5+#|%v@zHMpE;&S>A_2lyzoSj z+07nM8fS4s`Gb}e)Lb4_UKqF!=cFop{Rl*#$b3mL(d(>qQtPFQS;?$Egybw{yqQ?+ zCxD@Oh5Jc9&vvOdeMCMOzv zoDVKShz6QH^Af`rsjy1p>u4`wzAUTg{$}LKpjOH>LWZQ9Xtc$2VXAI(@R(LSc14|J}kc&+k zD%7lPM%!(=f$J3O%GUyut2Iwsoq6fKa|`p>!O$PK;UWICtzm7lzYfd)HZMK8kJ+{G z4ZXb#3nGwJ(LJaHCJ}q%KT1dkzrn(rrGoi}VKO{=G{ZwaA+aacrDR^AT#Bx&`a}F% zE6m{WeTj>+d6EqS^kHgn0H<0?rW`;Y>=hzh%B!9b5v=V8^_B~Ifvc-eCg4yOK&?zk ztj=Yl3%GQ6MC?0erz}kbFn`3LIKChKen|85{@*|4l#qAnlrWeG48TSef=dNZuyMH- z(Oe)8KlHAUp+ej7tlw-k zQB<)|+in1o-p2XA*%So=vB8dxAc7Tn85S1dv zY)<$1ptCEA07gl8^QHXQqwNRlwwExLDeeA>6zhtA+_d`P+nhKBREPo(h)v;<-P8J?YwV~*Sj9%hm6I)_ zy}c-&1fCeze;(0YL>$NX{6&bz-DSyobQwyiR{}O7^=p0z#aN7bh{8Wn5ZSS6#?Q`S zlu?t(8b&0VGw-4QoS1FhJI1A(`tB#xE#`p5Oc+I@US@(B^A13rU1-4DLms8dcWSL{ zNeykB=;n<@RPcEKn5Wp}6v;1aKAOz{?HwN{ommCJ<%6Y8r4Q$POMB%2+-G#JmN32a6bw z%O6Xy5*=8A1L9YVOQ4c!h8%f9^iSYrWVVL+0<8s86oJ+1N@;~%^ULL(#2cB219w{S zP|>a-=VKsYj%X+FGJ=m5kd$NOC^-!JGJ|KaBPF?XLAAt#fB-dkli`nU>*HJV5kuCKvmxzd| zL4eI#Z=FzsyHvAk;c$Y3GWxpe*GQi&)VkEG4t!O{#yBe}*RSvL&`=-zw8*mH z;%THFnhyvCp0=*~`wddse5_Cv6kzEnY$z8)zu~n-#tM#pPBwN32C8j2AT&*A7cLW^ zrkwf&V7qaHUi0MmaWsEU_nc5(_UrtDOc(HXPr$sUa^)UOU9JZdLnYrp2SGT*69l8^ z_lhOS%U>1xS<2wakgmcF8NIXkA**h8v*J_H^9BlN(imVAxSvcrR4=wOk%OA@C)cbU z77g8K7lb!`EMF5jC#O>;2za7ixZ)Daxwbl3@DrGfvbSJhP9-24Ld9KfI-0HhZ|8X)m%@Ew>=$h|Jq z`T*wpV?@5OqU)ddpgmV92hEJ<{lZlG13KFZl62V!4jwPZvX-vlcp{R9)gZNbP@=IC za_6E#ZB>4EJRuECFaZnTm-APXf>6wjK`}>w^p{9`IkAp7PWLc6qh%c(|DVGry9f-+ z2`1kZ0z#)FhMskPu3U37k|RH*?46Ie6cRjCBS&u}cF34A>L|X{3+`be)})A-$}%b? zgfSm4&F?qxz(D8I?+eU_SHhPI8mZ7Q04}|iHi^o}Pp|jA*EYbq+n>wJa{4f zzb6KRSE3H2mrtJm@fs>2(r8COe%!DLobPe=A4*~H7ZHYr$N9)35-pi`9m;{b@CB** zr^Heup{8mDSaXe#`T9RN#4rQXI**JjV7Tw&-&uE=R!I73#<|6v{sE=|lLap*SKEQP z*=^a`jx*M&*3W062Vw0F4CaM*o#xk*uc`He3kHI=C{pBGEb4D%&(j4*aLF*|9S}iT z`T!C*h&-UE5CPy1(C7~s)nb0m88fX1F=vUs*>>#0&&7l_sc>IJvr4CY2Y$+UxXxNh z?B{-2!%8ob6G1VMP_X4W5ozmOdCOYmF;ww&0r^BUheibLn{+9s^Q4?cyW~gFsBqS>^V%Opz-T+*CPjqBcp(hti)*jngBQsiL5^pVV>@Trz_yu zY{ff06xVV_wP{kRkdT(vb7hYfR}7U2SAH=~{-l+kB3*Fj_MJAO&xAGyN#2+kP~Q}V z0lrZUj2nTv%BKLD`5YiyI}bl2PgWbOk0tsT1TXKDZt-QpWkh=5O25LR8f)6#xH7L= z`jA-~48kT7V{z;TmKlo1hv3yTWQ)jTH^5Z-!s3zC$B{U~B-5a8oDxM&!w z+X7!e`80yCyS=svCGPD`n>-1}KEm0GnB*TE0Jj4DBx&H*&+2wmP)Lyat^&yqL+eSa z1s}Gq#r-{MqJ=o@8^mKAOauhq^^LV%5EzeH@sW!3!d}G{Qz;juY)ab&&NzWF{E1V* zQ(*pcXCzmZa#a_0TZM1{=?ZeP+@v4Ea_4L}A_c`a8#!P+sg5jKsPRN#%Ye%qRR~W# zBf=2m29Ah!SwS?1QO_Rb69&4Ngp>zKGn->_!9zi{4?zzC#;h>vlV9ozfq(h3AqI}5 zsGoZ}+?0G)<3f<9`@q@ECwg16ZG1DQgjT4Ydj+UDtp7;jf7RTU*V1hVxRTi$>nESO z>M`vyh3{W7C6_PJ<+QI^Z;GG)4*O|IeubN&lS1exDfh&9rzRYFIg*Wlf)b14OaR0x zum=P+7z?PK>h-6@>&+mdrIV?qE0evFSIkV+r{;pzUetJytj`jgot zu}d9q)1%tShpF^4* z7AZ%~p_o<5LKG|00q+gYa>a#%hA$(4`5GK+H#sOe%%GZbcwDv0+Wguh*j+d+;`+z{ zo$y2nAcsge4aKZxO6cA5;IuP>FC*6o1EZVtV^d?l@rm4gpl zjeJ%q5ArqL>nC*CIoCCMTrvaWoN@Pi$YejMDRy@&`Q%4H)T)3?ws($*{>p#hKck^Np zj>dm_*d~SffKJ1VC@yJ=fcxng|7e3fL66N6syuyt0FYpym!`dFUl~#asIr3LOGHNY zZE4W7eeP3h53);AVHn#JgRMq4{s5DVS-F_bvLj83T;+0i9zDibmMPDe|E+^6OOjQL zWt4qq6;P&Rol5Tvb2#VpSMPvnd_t$8pXgG-j|${2lPac7FNH>h$Rse53EzW529(*p zNCM6+?vyo#^DX?f6lVPdL8m?H(qr30420tlZPzlBe6l4R;dexO-@NPi2_jFesUT8f z5I|~FPS;!WCRcAZS^s0XC9gyjv_1vF*o*QwIAP(+YodI15Nn@V>wfVdHh|c8OxpiC zyOd)@KpYNZtnY+TR(d^0yBuB|VBqQp(>A>2(=H-^%o>OOi>77ZUX&KN#P3rtxq=*c zOB%=?(%`awu!;qMk1IC{0*$ka_R%pHlw(%w;1K&mMb6G_D5ab`HMyS0abqvxwBs0C7II*(Hg zCLU^IP#u}ZyU&IjwMn;OOgPmvLTOia7s}YB#daI!$`-$&cJs zt>#6=eqe$MMH5^iQ0T9o@9h7FrYAtKbb2%J#>pdEjL!Sa>#hW_$q5-?nWGaAPkmxT z!5KdPDWaNyvn3hJB9*rq8x9uByzJ14`p%8X1H8!S*5%hf{?l}})$1Iy0K)=;-1~wl zDAAE&6B52|62D=3??8N+|UOnY&@PZeoypcUhDUN zRNP|m0VxeQEeHdt<8I{UIjHMH6MCkZs0&Z{&R@4`VBRJ^B9>*uA6zkj?Ec!mLPyLu zy(s{SsT491RzJ;`pE4qhP#mS)Moc2u{10E_69dR-3k0nQZV3B+)$=X@x<59auEYR|uYCL~WE+>D_ z**XZWs{n~}D{i^<^Cki#1`TtCq^T&#t&YUcTj!2xJqN3HQ^*tO+>ruDO9DliT3T8J z6rD1pA%I=rFoY7nqQcm#vaaY}z_zNlq3gSmvi*H~hREPlmWD9UHSu#@V=7*{-RKP! zY6FH^bp5-6T{u^eG(889A0!659*4KlP7l?bNtu|CdMV$DiHmCp;YAhqPlL5XPWFEDZqqjmX_lvzGzCD`%*J=yrs6MdK z&m@s3`ZS3Egxt)SJz-O~PzLm#%;~A+xFm6^$SMX}xbQAz741`(^=jAJ9 zx%E)yLc`mc(U=(~qym?MGjn3)sITu}>_cBS^p&udgFm=H)^AHuvVXNtpaHoy3&1zV zCHmN_|MZzKv}~o0Nt1Gi1ERBc9jwzeA-od2{4G4G9vx&rjDTi5ZhmwW)cAgOW>njW}y4nM)7V9vR$ zQhVIp1MB$vW?6cANjvp?$FX~JW(gLUb3l&C^Ka#TzzuET@RZ!Vsdv~d15Mq^qbO&U zd;|>B*%qtua8KxRbuX>QCPhw_9h6h!eP+jnkA=SOE@?|S^n)|WcEm!9?-CbZQ*IJE zwnziuAsTL{nzR?FWI?HO34?Og7?wP85|41l@aQ-}CM;asr3Qz!7yZ4$ZE`hHK^&_7 zKw44RL~dmbb{BA3Vz=mIa*!-)M>fhQDDa+9)*blH9{%v3tua@SvkRhBA=Z{suXzup zDO15oXos8|2%>Ip>JQ*Qw=ZLDcV6s-$q(h>L-CSD0086*)<-@us02%amXH^kKCfV9 zMG&xbAJ0%{j+2+!x2X^d@$plzn6cO`$l15uCUSw8T=)`5(vd`Zz+@B&wN&)6c4?a6=y3ab)v zqU-)aNBL4wiIhjZm;&EE$h~sC9m&wlM`YkqB%C;4vFv zXaNN@5B}@G|Ma2>69Kv#Gsm~MB~Bhu!e~!nyZtoX-9e&w$`m*vDDQH4FiL&G7SM9g z3tOl#ad5bR{n&;g+bj{so`^xt!gza#-#XYL4}SW9foyV0IABD*;ChJGeE#?vaxtcn zR{+wJw;`v@o7X=Sf&D|e>HzygETF%l+h=GkiS<}_clznew>hboj`#m-8&d8BTTE{; z7F^|Y;;i0uqj}hW`#Pw|q#JaJfLQTUg-=o{h_)MR3s`Gn=r7ZGPQ4AkqY&u&;1v!w4~bfS#7{|kO^fD3@*y3(Tmqepe%esJLevmbn^5DopFhLm=!Vw+IPC5|;>wLiVK8tC~~Rw)7v)37*CboA_e~ zA~zfdlthptFZ^qE=^V9SU!jqKW6CT60M34R)#8EfEZ)-_knQN`@hVS(oCwZ|Q3o!} znAb38{C^r5FoUDF)|)>w_g#)xkJ0{aB9hO2c|o0J3jB8tVlO^LsN0L7Jpt9VJa)4W zN?$U9kEFR*N5zE=FV-CAGMk8}YgGZVASWk>8>uvtklf27#FjsU3!&!gi45L1;}^3~ zu2fHBNlc`m6Clu@xXbeHn`F(Jl)2nv0fkomHe2Sp42L6tc0D1fxp{xkPaYmPgq}Sy zB3P9}LdyL9VPkl7nE|}w28_q70C6{ zSU{`^0Myhkr40~f42$$`_6kbVq;9wjsK7uujY@ZQ7mX^@Y%sIm9Tc*)8uNdlc@q*EJ@1gzaEHi*^?D z9bY1fDn{fK8c<%y0*nxkM@@*{a5zo*khoSeDyyvm(%?B#EL~Zw8Sv5LteWm9IB~ji z0$XY-l;nbsx_j8r2d7B~9z0(<<{Ib+0+;pBHnR@&g)z|5?&;`k;nAwcyFag67wJ$S zq8IXc__I+tHA70Z;1hCc6R_uto{oYb7+a@RftK(jaE9c&qyIyb%#JU;a=v?ztR-NT z^ae)zFYrnLN;9wz!z_cd9F~*(tBxq$!kS+2t5e>daRvdtyu_ zN`P<9kyB~x9#>L!D$Ncs^9u9HmXzKAee8_h3A!Fk;A;zkwQK~^cJ{1|V99!;Zb8tP zX}!T?_8OEWvfcpDsX_m7_ws6affRrap>07TB`F4Q7O_P6EyDQ_Z$CGQxYsw-s04)U zb(%0bJ7o>kfO9i;I&-?Lchh8AB&eQA@AdWhcE!Ze36KH;U=eI#$xvUdZv}jQz+VqQ z$6ESmEQ7F$41*qqF5~~om}V8hcM=~qxC^mK90AQq-SX>{hPl4;oPJ1zs6{-Y?YOK= zgsfy()$4`6ALKppzSy7L^HL^ajRr)%)cYy}vc-f9m@^~XtW*~r;}*o?(IbQq>qEkA zNOZJF&3LbdsAacq5~&Zsot!*@4oRdxq>AyL4wt)(bZlcAax7L;SyFt*i| zONSzkuAt&#M$n=)IThL?yjvj}InRIm6bMZ{l|JKIz~wg#8m5x&CyHBi@30w83dy@_ zxw}x|`GsF1)dm4Po8EVOg$Dv>OKG7SebIem+2U2!U?J!y+MS$9SPkI1jh{?9%LCux ztq(G{bzdJX#>T}_d|GUBwnhfAymDT$g%3i;Uysh`frlCo`18_@{tZmBcXl*5QK#8FxrAxXSVd!oI1i_%ByIY3_K|oTvq@-I) z1Vp+!q+2?_Gve>Ad%v~b#UH?$dCxv)@BKVG&Tf=8miPocY95jBWlt(9)smuYt|2l# zBz>>&ZrUFtqE2p1l5a~rKlcJ1GUFOHfCGpjg|ZQfee@hR0|hm>3d^lkmZxK0ey4Yz;7I5%PEKL6e#&~ytajRilTU&Iibz5ISh}~nC zQaJf6P9G!Xzo6wiu-~uWt4~Yl*KRv_)|%h-p8A9QVG$P_WZlC!P!<*%C=0gM&aJ%w zb;EzHtXpu%@PzTZ`ApyU4&+>8#W^|1M||tGN@3t4-fy3u29%oik(bCSrHPW+G@a$>c-__`0(4b@C`okB zcyj&dWa$^J?)hrHZnY_G234Ce97D=A7N3K|{fJ*)Wdl1euQa1dKC&DAeu1S005Kf= zKHoHxM>eTky?t;={%(!0AcE$}kglVG9M%goZaRr9y2QurkUlz;;E)1hO-+3Uc@*9! z^u4oG945;@qLmUE)oGLxxfvhxQK$M`yMa+dB0ERrU%%RR;OopE4yni(8j^co?#ndX zTwgeC>6gF3c!xzX``xUT^xskV5d%c)_Pv8b+e+~CL7dGM5rWD29~jVdBw-V)`Hhd; z&L-~NpMTs-X)P8TKh|m-{M0!3QCFEce`Y62*rtoXy*H^)n!=jDg%BWL3{Lfcc2+Hj zkND!Col;2*pOmEQO6%B=gTh-g@oHiTmS+UI%VH?>Up@a&Xo!`GN zf};&ALCMQLFq76#hTN_FY8DJ()|dHa>!qrt6*N=tqEN}g$;CBLZDRzJl1f=lnz5Eq zQVPHD3Rq}=_ocY_9k!tR@z%4>dulKtPR_5tgfZVsZtPmPH(qBQs$GAT8SXWkYiiWt zFrmLy!fk{Sh9|5hM$V5lGMCz4m|WV~vb453vVVU^&7DEa!LiW4`KeW=Sg6Iq>2rnz z97~OXxg@CfO=te6HjbQ=R}VKu3CwdI)+Q>~-=cgZa2#NFKHJNKFVMo)ZNBoYXRoxC zo%S_bqk}p8;&jq*2)sP0U2KPg8S^i}3=W*TcFZO>*H`od{LWiM!ZtSWbDTx#pu%8D zA4=_pz19cx;bbE26^`W@&+DCc`b4h|H55~YC$1`Gp*&#JV?Ma21I!(1`}v7$ChY-- zINr$dTM2QfL`b0s7zLidV;Ogt-MoFaQCMQzf=YO56K{K)+_Y5A1nwB`SsZ*gV?-T|mcpAD4>JQ(QzW*Xvm1L)7LpLvZ}CQo+)h zOB!$}ybhz2emadke-GxxlX#sU?lZ|)KYR8F*s8%-y}^fvj$d@kkVM=Mqek=D!1Yo5 z*h#!r_EV*=x^UpbnyILjSQMgRVPVawdiB@moBO|nc56BzPqU4hy&-`jfPvnGx8eUK zKV1m&BjA2r?q##1eQCEx_+{pVrHbNNTt+O6nC-x{AhEtVXV=^g)u0`?5G2NjvLx`)0@P%wD6w7 z*#``sp@Q6%;ErCA%ld%R@l8tTcdSO%bp~oT`E1zU93R?koU9VD9aZr2&NOg03?ka5 z^xnI5Mbfg2%|*y!yOO=5{jqRsFt%5?qX!iy=fm`L%IjyE9p!-S4-bz&wie;Hiq0!S z^Q+B3TBx&iRN4%DcYL|6f%dzJqAPB@Y%#lCYKR7FdG6 zfs4pH;442*_u8;1t=ZrN5DtkfZep%C552+UG&&_{H3?#P)%te{1B2vYoh+LTFhj>9 zS@sW&wYcNX^rP(T?3(k9!ZOEo^^?`+t7DRPm}HZmi6~JXF67o+Jx0N%Ze?>Ftey<7DnlP54GxcRs+0wZ5 zTcHEy;V{4lei>}m!WV_P+(BJJn=Aphtym(q8 z1hG0FGbK&v9p%=;n;y0G*Izw$w3(ymD)Y%k(yCgm_#1VX{al1_k<5bzh|&#f6k;aj zlsGd93;(iS#cYNJqu~%->c9(BRMf$6f8f7)6(pwNVVB@P;SmwV<=5-EX@)X1`9(!R44PL!1f#V)2nY~N zzi3eoK{OsYHw$`|zQDpH*lN9Q*d9;0xQ~gy)3X18IxmkgCNSg&9Dv;v z-(rF~cKX&f$!z}#_eM;?6nkr}@oi<@B4@b&U}YudxD}?{Zwq|b~GIpFj&-4%dir*1J9u;hKIdR+Zu z%R4DQNv}#spGGmPk) zDwo!LJZX8_DR!;ac|1NoE@)J*&KS(sCwAlM2@Vx~FjT_8b+6^Vipk!05}Wx^eH*9d z8-R{pw8No;MGzfgh8ATwb_VKXYMd2X#BD@bRR}@cBCupBU>WHTTEjM# z6*qZHJh;f&PucW9kk(nU{#4nFA}ttm{}a4l1{I56=eGQ!9|gQ;B7_T3DtPBHA2h>T zRaYhBXlmG$!(_bFShw0bMThymURa(D%-Xux@#R7<-(2kV;oFXmjsz~#h{J=b;6nv5 zU}fY?+QA@UzfdsTJIlO19K#N`ZG_zGP)+V~Pw~?sfMIg$pJDX?IUrj^R(pS?xP>TH z;6(RpH~e0)a#CfR>#kY6p zj+MN!Fl6OStmfVdln*uRd-H)Uss;klMGbP*PWs+O&Z@&cO6q+h3H)!_%TCM_nZ-E& znv}=tY_*dt)zj)S40&Rave}McdRYpMG$j4yaGn~5dh=RF=wo!fP+l+}4uQB5IA4d< zmrT5M{)3oQhHS0UQ+-bc5w!RMWC&m5-#9W0LN`0|hD>o(`R->&N`pfJFR}@! zT2h5#mRLxLh44vnp3K50|Mf)6;+6)tKiCu$Y`vSd{51RXAOcAf0GCG=s}S^n30 zf+wZXO&rXBz!xRv?(@ML`Ssl%PmArT9Fssk?2hdq02QR`S)bS85gORG$ff1x=28m_ zH?&ZVVu@X{zv)Y53D5U=DKTA4B;@?0%4RM?#N&AQlTO1JU{Pu95Dq_CvP!do3_*ur zCort~^B8c=a|Fk(Z7E*2?yo}9_kHzba3z88Z|_lwSdb$JyKj~RcHgQqZ8iq99^`2S z=21}OSX;PvGAV~)??8;7PV4UFfcZn1i_-qgjWKx>{Q%Z3FE4WocwS$gsW(CfJZm3O zo|hQZs^jnrC*^7Fw?1G?u6UzAzJx)4{I`Y)rymm`B>GXkz1WLq57crZ{|dYcYg4056EXXum|Vy~wP-)Kz5iFWm-Rvo5T z{tH`?0kKt)Sr+5DB2D-Cng09>6@5|s5w)qdXAVx_Mdl#fn#qDJ1A)URnKrojc`lOq zNsCfSXfNBG-fJzN%M~*ihN(K(uH8~MkPY^~gJpyK^rOSQ&^sp%r9T{=|Mh)-Rfxi# zB3TgddF(fe9aRXidn`o4tn;MPZ|mxV$II^#ZEIyqdP^3eonKQ?z`u4(y+^(kl&ttC z6j#^6pEUy;@6c?;98CYwcSIzsQbxD_`dJb0G8v1ww>2gDpAfajhP6@*Oi!eq$q6uZ zGpjw<3P>b=@TF3F{0lEA2j?fnVhYY=cN>MEH>J>^x1~nAd(w9+`U|)_d zD+`P^`0g3*{6f*?OW1=(`fpcTBFx8s1MN7fbiiwMhmjOrhlYNd=k08YW{Jd>$;=hx|4)$f6D2l58K$5 zaxQLh{ET&cAo_l;Yj9%m;%`U6`9NIlB>Xty2i-kPFTz5l^e$dGZhVN?@%kDy!9(_J zMv!V0udUxr?lT9Jie~_YQW~X@heqOXdu+@~+1pUG%6bZXSPkufK;Z*H)7O=?wMRB` z5abF~6@L?0k_`{*ZQq7CQVZ?3wkbri|C$!8%P2KD{BK}c+#Aq~A8d3Ld3H#PE3FZ| z9;H@8tZyf39!%6@r8hL-~g+eJC{YRdFUcn7zYg_)!9mXU z@5E6fnYiZ|8^PYl9vLn-JA=Iss+u7`v;wgSiuKA2hsVa}V2-DKS-@+s*C@54l&<$f1;k4T@zp|bDgxv3AYm;4rgDo^> zBX);Ux+byaE_5piyD0?wsO9ZcU3h*H(pP=r)L0W3N$ZL_!u){e@qHxWhL4hXW&zC* zTd|8!n$=r#6~zorfqi(G^EML`LrX~^pYf4Pv&mFiy>UpWlw<7Og>y)Id%LXp$KgTP zWNt_Z`mi1g7*!rPVGz-jd;IOw>ch76%zIQE0){Nbyz8B}_Mc4%Dv|X{kNpOtt9Zp) zg8TPp19d(vO?}m8R!^;?C#;Gm)%n!I`r9KP3jXam1TW|VonK&a2}lGlSIC%n7<8K# zLliL9uq}&=;5@9eB?_0$Zwk;(%zLV)t$nYskpR^VbuD4! zuBi%9M0qEcjJ;mQ2K$d^f!ON_n6GwiK0?&QUf3(m&}?ajGCbvwa1x_i!~W}q@1Suj z0RuQqX+ibv>PwEwL2JI7X-yhWGb-e7ZcwP; z_)&kmBIU12{+mOU6(>jN7d56{sma-gw7m(BxzFs)FdPb#mtp;zM|!q1C8Byo+M1ei znP7@vFynrWz-@%VS_hUZ}1n z`hndiW%!yLe``iq5zC~-+4$OKUA{EKC)yTI;T_uE4)nC++Y9!*9=S(u8qb~ucWPQ_Z9RZ^hI&~XB{#S)pFf08sQ*FEnco+rHO!cu`a6updV4oAL``Bvn$+jKqBmbrBCXg_j>FKycdb^vBAU* zN=@2TUo6)xDE<)&%#vG&WY?%?^D=$I)j=mb3=RuxfdgYiTc2q*Raj7?@&p_kcMb5%LV2&uP?{e)I7kiXxnoSg5b1?C++5C!_KEPlrLIg`!eeW&*JpKP|^a zbP)1h*iPknGh*Sm z78Svd`#__I=kpr}M*E1B`Q!&O3&-zmAT^&_@Tfs(osp4IR!&QL|91{j+pyBCl|E=A zv@>D5)#?B9(<};M^~1ygtcS|hZKQa#O|m3hoa)2F!eGf9ca4pWH=ML-Qzzc?#>U}i z=H%RS`BjG&@IDQPz0aRgdYF(h(825C2qPh~B~h0(KPm9W{tNw64k}%tn9bY$`*)~r z364MbGOa+hDrHdgYi0Jza(vo{^_$AvO$ZpaIQuTjDwdeabGl`xm1zAYZ%9YJzAJ`Trj?U&N9^|65X`kjJjEu+vq-n{y#KZtAa?09CK4qjrL|15a;{)<24N!b@LwkShwKpV(f<%Zx6zymNw3a_ zKDw@V1^wvWco-91NvVgbAsx{B`pqj-)1~A4Htnb}ou|ED#Y3QYl**X68hlqBfhK`- zP2mE!l}jL87yuCBM<~7CI@#SF(_Q{D!dqHq4-%WI)?1U^2ZW|UndUStT!fBv@wU2IfVHnX6H%UEhfAf}Ch`iy*e3^nx+TH{m6#M9d zVxR6r-?jzur)r|=leF{sW4kiCx|`~@XZqlN;)Sc@Xon5D1*y-Qa-e{~yQGx zU|%h$wBWuNz|+&{w@2Rsaoo3|)}MTW&N^ytR(vQ?1oK5vl6WC-K^*1r{lg@z$BeY; z9m*FPVYX;cKqUwUCo3(>l=MjQMuwFDhp8G`XSJNiyX1w z*kwF)DDfFY|= zTB$eH_pcRaU>vRQ!&{2~*#^-2B;24SiMJjAI0M%S((PISO@9M+Xsgj@!UJ<*hvW@4 z3k$|~-Ix>P35}0ntk};Vy*(X&Ldl{Xg%R!$r^mL zL>RK{kNRmvI_^g*@5uk{El9Xs4oa*H03h9=k65ge!PmOn<^_Ex5y#LYu<_6C+JeE) zWWfh=!Ez1I1u{;hL(ce5YPy=fZo8rpm}El@#{&H(fj!tU9C7(c4H4bJ5f&_=I z?EJSdN5sP1QOuj^!BOA2e=FWfP?M2<4B^`~0u;cfdGTpGk%@6wxTW_U53>+o*$K^O zS`lNS?&xJXL?xa!%p9$+A8PQq;l~N40CO)#6sKdb5|20+o^uaKZ>z?|kwWOcwdqT} zkNiFcaKlfQf5qg^0Ll%MFRWZ8ar`#cwcLVp(Q?my65*EnJZymE^@_ppY!9vE(#8^q zu47oqFlZ%9*Oa~W+iOPf&Oo8$g33|YGNIvtHab>EkN<+zsj9CA2Kk0c<2wRG9_ zlKpp1Xc8%*B9Ic12HzQ;UVb?aEO-Sam>?J$O28tatQ;}c?tZip)D=Vja<0ki^rZP_ zo1D+29Rfl}vzu-W>ot~RtWWf-9_m-UZOKte$|~5FeE0Avh=p;0$X2p_HzU20!|9%J z!_lbz&PhqlBKqO_kj(jc^UW1o)8&D}i78vd9@65{QX@yuzc7fbcpUKg%sr+3c?1Ad zSyS?3-Vza2gcS`z`)GtQbKGe?&37f|xH*Xt9UbkRk`$94FRrA7alA7XYPup82!OuX zu$llI+FBDEO9Kmh-?n4>xu!<5(Qm)p6k+HL{$!`ZLfhdW&SY|Req{JEnct2|^ybpR z^J2R?f!)~OW7_Lv<~@j2xQeJ~kXiAPqUq#b4<#U2*{OchOPp-? z{n1Vd2wZ3w*muzmj@?|Yig8$uJ~=U!sQK~Z$6jb}lk!)O6KgqAyUNYtdQ9Mjo0QwG zZx1;!JvbTRHg`VO_FaAX>N^MsKJZ+AF^B-^N>0t2)@@J|Ae9`9T3A_&theK2Zq8U> z(Y-(WHJ&&Hw2Hz^B#=TvLK~eWvaCZw`C2S0)nfNPB$C?5X=;vq&X`<0{%k4TEtj99 zNh!^rqAa(3x>LW~fIfD%nih#AdU}U0MsA}7v_ZpZ7T*OIT20re)or)j)?k8;J%r_F zs~FeW0ha$aIpySSm3Y4%&e{m594D`|9D6U~kq~BPXT%>b9I*(qowz`wcDQld zyQd{%d=|&gOjBrQ*dZXocQWgNrID*#RPL>&1u> zl~Vc_i;;6x3u!LR@Z#rnl-%5!`oD(so1wP+q~p~-Y~)_on91&Xb1^GNxkQMl%Y&6> zzu*|Uy0P`yBIY`$Z7Q!umpSjPiTXavxrDkdOHC)iZULv5RSdLdI20crpPlGo?+5fi zBH%B=e*RR9jgPM%BU}%i_57n0`2KzUT#}fNsCCoXiXd;uISxownyH6juctkUYc7uG zUNM!ELqkLor>70_XJIEhGwDO?$|fe&Q3}{DS$h%wcRRPc)oWNgWt(4^2nXlC;nw>)=MuM7wqFkXj zyL8gBbgj8B?NzK;2ZZAc(1JzrV{dp}i-^aOW62n?nty(h=bVDQ;!Ldmt2oj}qa_a% zIn-1ei1si4^eq7`v6IPl*_|D)bFvKB*#Bgj8l2{HJu#yL!Q-28j8QR9j35>82btLu z(u}B+3iBaiMX1weQMJizgPW6exbVRbYS+bJ`mFtonq!d9q`lB7($g}1KV#QP=DLxe zi;a&TlqnM_Kld}$KeVts;pO^ZHj#jR)~-FlXM(}z&tid0T*KKquitkO7~We*An7BdvYv-RC{3ZKg|8e#Zxza*9%RX|}d((u{Y z8OG!$)1KGf{8Ny#tQ|BNG}uN$G;#nu|mlU@ny2J+D-D5G&-ZaJ;SCo=3Db zn7w3%w>HM~)jfK&PL>_J^p<8G3rR~$i~oZIQK*k6uDe%)N4Gc}YGNP#Rc<;8I|C-^ zA6UXeb1f*QBmr~f-@?5Az~G{kV>zm>OG`4t%bxVyN$Dh#Xdk3psi~+kp5o=Ex--FA z%m$F6bJjs!NM}j&^@iaEh|PlVX(pDj`Lny0(H>SHJLWV>5Q_PT6J>L(q@?^h>=D=A zK@Xe4?g$~Z0a*usG;cvusoL}BERFQmwxs;FcO+Epc*?xh)jw!PF2!mn?{}Fx|I1Rb zA_CKV#}ijfppWdD<}taWsO#Px(VN$ITU(Wft1$wJ zKqT*Qa|{dctf1AHvuiwBlb0Ck1c{4_>OL9jM+Pmx%c*k*Ybtp+J{r)>LSmGL+vJ?< zi~)YqN0Lix0`pP>1{Z^f+bqgs`}{^1dl6AX<@teV|7gsp-X;RxlFCEdJ|=d3N-Z5u zOxvr#(Yo4i|1Z#j3=B(;RXQC2>J}k%h^9v1w#?vwVIA#;MiQ{JxEOtrmRHvkPqCU~ zMCGEWuj=9v^}J$!WBQlVIlk7Ar_Z^s&qEOI66A# zTL`VKY{~6nb*ziOHLrU4B?r{1r>Wt-TWsi2PJYh=l25hwn6JU7ibggGBfm?)Me5sO zxnF9wliPcavxa%~{upME0X^#sE&bDXAMtO6pxeb&eX<>taT0b55VG}MK6YB1fERg6>0FH!oskf za1Y(R0t`d(l( zOS6{4n`{CAKpBVnYkB$RaDZ;<3%}S8T_XU>;mQ9y@_Psp)acvJ8bS&9V}j~{>MbB3 z08D&9!WNy_dE|5on=r{YAtAw6Z-$Z!zq&7Ra+&zX?T_WFSCws#mXw8UH;ah&m-St0 z)9MJ93|-Bj2NC$SW1JnGi$JXA_E!78`%Bf0{u|xI$xgH83G8P-=`Q%_rL&x5TT@gF zc2gy%#og9&2$8zSle(TG$r)RtrFy;ae+S;OAW)IEx! zHyX@h=ugzgVdZE=ZPkc>fzql^$0G`v?o4uVsSt`^S$TovdA+7}Q_`)9IiB4xoPP z*1t|Q7N9}y3L#gq08l?eQ_BdyH4Xo|K9*2eB%|)^?jb#XXlDA*t>cw9NeurH=;jj zXF$xk57+%dbHe;uh)^wyFw&mrz6;jql=q7BZkTL5K8sHaqNKi{0iH&qI zyi`2Us%RGDMn#je8{c0Qm3y^5Ey-?YG8fUNd)^g^%}4o0)SG4UHH|{pmh*ji^q&bA zKc=v&d68y96hivb9hk_IJWR#g;nCDG@~o~78tK{jNp&upA=c09ho1t@Rx~Hzr#O=( z2#Sg1aTsHSlNcV)fE>QyZZOiIxw(_WZm>QMRj-DEwe+;l%@v~Z!6STdP(2?~Stvk? zIsTUSWQ;2O56NTcmJZLr_um~H0_(%xZ%!w$GLc#k^My>;+udTn~6vY z^x0lV<*yB7nr5!|U76SC5|w!k8G3%5PZAso4REy+-w7q6ykeR??wEjx^sJ%o#Aq!{sSio3w?_XC!F_wTV!VDza~_t71oTVMLzN zMv6^#{uOz$NwJNakm3aIro&2X;aRUUh4tN36IfA4$Y2KN`V~A6LbPO(h3sxuX!6So z7_}&i;x~bnqap8Yyvx)~MLLk}a3zMnt$A7KL755Nn?`_t@)_W1ILxympvWg@w6J zBHdUoq_3-ig>xpEVc}CwmlJM=z*C9wh49CNOsiV_<8%2*uV|%RaZB3XV^LFQu2Xa$ zbp)=g;G1;BvFJyAa-*U!QUKQiY*-+Q!#$3-?AYQze4s+Z1f{3}rgt|1y5E|r#sZnj zA{=U%9z!&F_lWz+4htH`Vw`lf^-Q8e(!Wx=C|?*NjRpC?X)M2Y0IaL-Ho5K$@Rj}P zdrL!0R~Z}wE}OOXdCMf|DF1Pbup=S#kHS9CkL()y5Z4;gQ8(SyY&Gq2C62vskcb9V< zB)1H%N`zr!Nu&v_!_YIVNKO{lS_~ee?3@I;Qd!H?v_tIl7t0>eqf0$Vmxl0W)P}g< zFMJbNnx2=3Epq==`dlEcc^MLmbj{*oT|Ovs_1oA;ym#WEk)M>v^a3I3qQ`@DbAjl3 z?X=iRN#1{R#pCu`AyEH!HE4FdJY%k?(s0zJ&Kt4gU&wU@Oq)wdG+Fd0I5^^ZNJOe8 zZL@3qYy9IxQ)`c@1f9i9i9BQ;Z8@PpJMHfE@G}WU1erXN`oi#3P8>&EzFR?-{>sZT z&2_PN>7&D3W689~jcL)BSeI_mX0O@jw%0bhSA{!GEpVvt>1IiDP8-&)JO7ylnCWFu z`yoR3Kz5)*_7x1|ZeG86vr|4Cg3k;SLFdN4a~JWbQu_AjVL=EU!}@MXbXNC zBvuz?%pV7%8lKKXqblt91o7UK7RkJVNj|5xy0D9v-HsK>qj>%11b>O2bMf5-7Cf!e zByctnnu+x*GqhMl<7b;mJ0<}Z-0U>ZXhYAoASWx9rf$28x!p4PVEwGqkWdVZL;%j( zeCD~&#kC>!nSu;l{*fkXVKJ`YVt53rCTi)ZqG)i-U+Z(CQ@$JlR}Gk9Qzs@SnyEa; ze{MTvV5|YQ9ZiK+y+tP$_IRM>&r!jKh2F&(+b74UI=L$jjj!2(&fBv}Vpem?YLwEX z4nH|6oVO=;$XrK7_9`n+=6urlMLwif0V*(Q(|i^CE;cT15Hv2B9c_#lUZi=SuO9+} zHB@Tc=HAc*qDg`~HLOxmQ(zvD^mP0bFnCG^c>2659)MFyj2Zceuozt{ERapapL9!o zy^ldWERwpqDDe8vs}oQ~Hn27EJl$wo^!oJW;5z}4!+uE<5vu^AbVuQSG!*eWDB=JY z)HHOcmEzt_HWhNi^#H@7!FKLR-!Tak>65Zj5=jv{LT4h<$F9OieE#kH)O=}e*x@P{ zYp%kWS>$5A`EoR@=9rarBbkN&haj$WJ0!w)UKZjLp2;qHvnQlidKj58RcwWSxJK3& znpTNg`7BIUUT%wZtv4E4{B?s|V>iD57fQy-nekSJR>X624whzyOFf>`iV?(pGRw@! z$Rhv-B9xR6=g%v}zGz7|YHn(e#+h7Uaf(4C)o-38g*?Wo;(Hy-(oO|avMMSK2`bEk zQfGt>iLS7J0aZ|gK-?-UEG)5UX>5Z;7^M8{uW}qXwlXrPOv?O+#O%HHMP?VBIUW9jjs zbtA8HG7-mPneokU_ldbhVQV5;PaZ0=Dr{?tcnX7YK6}|6AjV&MIv$5}TUA=H=daVL`Q!4T{ z{%x5Z$BY_Ly-&es8EcOwnSph__@a#SL}|g7qjIsa(^@iq9gx(e2C-x&Z}2(yua0I3 z^Ht_F>ki*JFz~WQ@OocRH}1E}&zPWu5$La3pKh_h+g7>g%rdb2~0>9K<@GXD;Hni8OZ#>s5QCM zvA1n%%C^rIIpRnqizCiXiU!$e2?rHjEn(aiob4=BGG zm5sV8X|yE3ARRQiJa5JA3Hls*{6gTrS^&Et$@t*82hEeP8`p#QhtxxTg3)xz_OY&3 zXJXm|iCS|gUmmr1x!h>VMNW_6`#qjY?)=X zGEH5>6$^zj$AiWZipFI1Aq|+A@)b>}`k!4+{Err|0;|lN|IBLFT_nO;1ldv;WStNWU2vB%A^T?rO9k#4%uY_X zU`~<$DsLl=*hWao7a2`FtTcCMwa+6=o_8nf%a=X8k~bj%?_Lk_xLg|Ex!UDn@RgciI)U@%7oVNSrX42?wC_QZ*j~w#*iATiJBBc?-i@M^>i^fkPK6f zkMP+W7nkBr+nwxYg61Cg2TQpWa`ZRO4MwXyYxk_^i;uqXP}GQ=>^0|`dC5fzu-5Mz zbk8&wI=Aby73ZkT-X!Lv*a?T=K9p4sN^^Eswcv8d$&XVtiYQEl6!J>=P=aigVVj`2NGT@&Vq zn*zB(ieWGAz8G4e_;}JDjXrTzp69NDp`|l+N}8Kk7d*tNrLzm=btPzC=@OAb!+#2M z<04pFQwfY(kqj&Kr8cHQs2_NA{eT7*&C`zO$=_Lh_7U&g@AlUp?|^Bv?!TI6bW&X& zI8_{G0GQ!n-A`37c=}@?HX&KuATTjh*TvBhy-XaPvE+%HOjg#1aH-&i%NEV;g6tyT zgRWaDX7?Kzndt)rg#`0Gb+p(uP({A;Bj&+#;!l-2@qksLVjfRW`oY^ zt|b8tC)dS5UNp6P!hFTB(h~{W7$?fhGh`Fk=b?}>gmw7)!~*TmM#B|+7I^1ZxfCfX z*X>{Y>F$6lRvpzu_8QD})Ng*04&Mncjpx@3*c7v!Qqz+@stk|yQDyCDx9vsCs4w=G zfZ#h@^!D~vygX4(^qBjY6AaC!)i?W@z`^bQ)67raG7IaorS8EL9 zP}QABi+!S!-g^`O#1NM1k8(3()c8(3DlEr(@vIUlP#aGTN~?(yE$#t-H>Rb|=~j+w z0^@eY;=SErUtbJ_ExD)MPEdz7FTBKl`|I%(u)`kir&(>kas#jg6Dn@v%2t(&%9x9} z?Do`NGEj0D;!Vw-ZW9#DXxLyZIq7vt^i$t*sJimOXyzvGZ%;YMIh-2287Rg^ipS zQjjhrlfa$Kkh_oXAl*K*;8h7$Ta{64Q@59uwow0y?;FWoIP_SUPykg%SNXK0DPP8MVw&%w!`2QI7{X><~4Y&L^t)cg81mVQxeT~{9D zE(g^(ZiwCge48N}__q6P|M%N~iFNn|WV3oINbEw)%He;q^N7|s@cqEB(lxqnVOFd< z6c)DC_d~ZF27yDR8+^Pl_w%djuCVR^Fbmg8Om=rLx5kSJ=I8X3wmx!^a`PD&jgP)1XOQ&4x106PhBM{qV z0}8!v*fZqw1P(;3uCFfylpQwR_$CzFUQ&ydu-1Szk?VxBySL|od)n_2CrxcFg@AKY z)s`PE!#%z9*cks*NBkQHimlrwk&ci_CVyXqEf+5uLiQ1KzT8U3jy_rFx(?|p0M^~D zENcq3Wijjf^V>!z{mF}7DwwA{9c|W|Blea&21-;EE0Oz#;sB}XZ_wHJ2l6__$jj%; z=31jyLRviORX)kb!q^Ga9zGIonX=TVoe_*<_7I3hZ#RcwiyX%#onHj^>Q?UNqbe2wMBGnrAI~iYG zPKq~ab`HIUhX?(~?Yyk`v@;;vor~bJb<-4CnJ_Wgu&d>(60;+H%Z|SLWYH=^rV%G~ zMa?1yHcWF`uUF#oYimC_Z_Ua%?A8y)CPEFU7h@_n%`SsGhfah9y(YW;VG1wtE4gbN zW#n8#?x#xSlwX*t9iH1m?~|Ax!{TnPG~QyjKsL<J`?u}2s z=!WaJ$HW4@lev7>ebZau1SU0nGJzRhV+-CK%&E%jaRNS2Nx~(C$#t4H?%Kh!I@Avh zXtz)D8JC_w<_bndwbtjg&h2<$ltV&Z6MOz92bcca-05~CH;v8X>?1j-mv+NGW065q z&eU{|jcD4t@U6tiB<--8!@gD3*)>XuhP>h}w}H#nm>v(9NkW-VlcVx4y4BnGBjkP;a}fnq_U`Gcq#ziB+Ps_bV$* zULKu?mp8;JAv#gc11vvbzqo@GVRv3Q0OfhR^P>0ou{3|@nDY!H=t*%WxIp?#Hlh%M z`}3rs;!w$bw=$Bk-X|iTgjC4dM{wJpk;)4DM!&f(BaXLtPU6OGAGz$;-S-h$T&7C6 zVqzElg^r7IGX3LrnRzarHv4?aPY+kVYH1qi%A>yDRpI)TCXw&;A-_B`rr>$y;bO*3 z|F5FoqE@0?2i8$z>>4C#5#oe5$Frxf;nGikkMz==+*Gm+^>D5pDcEH5Jl?YiEp(hPpk{_N3v{ z$l2?L1fOfJ8yP#hvSHORn2XULnwk1#_eEk-lIAH1ACF}oll0ZT2W(KUF`s9OH=Y|x z`+wjZlIA-N<@uS70VYc(9di{X?G4R8(|>L?9j!+V?Up-BY!fcmKwK_HkHAhQbLJnK9@GLyOR4|45Jxvf7#!A+;6{A+pz9bVIdJv`L2EpN=DWA z2VJNDbYXa_Cbm|n(wzBz*FEkY7O_cU(ewPEUz-uK{mKnHHg06pDVQW%`qhUCOXWaV^_`&1$U>2#Vj&p<1fHQmj z~dLM1qcr#f5?fhvtydDW; z`8F4AnHO+q@d2^gt3(Zm9Zg5f$z=?aiw+v4Zzk>O(B;k5m`v=4M8z0o5xt*{7rE%} z%T>i@cYU@)^nGL`|J-8=h6?xn49y){=yZ7=QB_re#^F2Fd~;0hwLjp#{|DowFT3fP zS9{+Rz0&ZvC#_;^>#W2@gEEsv7E2|+1pVkz2P<#-#4aNYJ+7~Iim)hn5>~Fy-clyp z7M7T2c@%Fljf*VAX2Kun@bPYC*B86{6Fj*Z5;bY}&-R(lktyw4F?=!xZZ)pc*5%jgR|&G%4Zw7%&ZPCE-) zEB*2f61}U@$sMt?y0uk_Uk8h+LJpI;tK?_ZHBrsm`^=02v}GO>JD!rj`(m=$$ocijpq}BNzAszmC;o~pM4})(d zzg;wgeEQHemYT%^`i|NmTn&)|whaAUWZcVtQj^0RIsJPif&$^Vj)jKDp#kK5LIZoIKv1 z4A003eA~o&eseIL<{nmHYqf#%9+QS-d^qwZ24WSMP*{G zMUXF4D9x2jDbgzn89*e{e8%x7?52l{xs^$ezEL5|JbShBN>$zh=i{y;+EaBbl$DkJoG`AN z7LA?`Xj{>1XgF2iG73j)MKu}St)GOz?!+j^J$16Pr6CYccMOxIo41S($2E%{`qT7`KFC@ad%()8ignAqTYbK2z6u4maVqpyeq*=$Ko?Hn8Dlq<4`)PHc0wz}UCrlPErp-+A- z>9Tz1#O-RAIk2B9Nj^=S_xn)*S1mHC~00RyXi?{+w$HazdX zb+wq?NBGu@1%FFvQtyUcZiD1s>-iq>1O3qFs5N6gYOs_P>p^LSw=)ef*Y3NVLzv8C zp*80#p?NtjI_O_yfurLBhpp`KK#9@Zjv$V3uIUDyoj31|EN4fIX%>t|gKwM9?2gyT zb4o4EW4x*`K|OScaOA8kq^}Z`ava_Jev9uJ5Z;ZG$q@vo;p}&XOQ^Q)JOQ^b0+fPn z4^WCk1b@bZWc&7@*fg<3_Zv@vx9vpAKJ5cgj7p`9H%}E6+0NT~S3~PCrpfEv7{iIU z6!G*{zNgi!_FO+3OMUthlbmvpZHE2AoAQM%8L~C!f$wVFtJw(&hP){aHVXw53ns>> z3I|1@E7!!!q++Czw+{&2SB@`VmJDHSdgFOJsJk?U*0_fNiYzg_lT zbRpmw1>iyppU1I#0=IqcRbBo%mr52IJq7Lc_m1*8gDNI$i|nzf$;rH$8u;w>!S3O# z>GS+*Dn@D9Ovz7m!DIcSN&V;qmwH)1@Yw2lUq`q({DY9KS>t<3A(U5091`nRLCOf( z-9dT-Xl3dnJdXPiUvYk*;P{x`y2!VIg%1GW!9IAejWzwLdv7jdj(W5?)wgFtXe1VM zOd(|Ob7*2h?x<2JHZ_`x^(V;&F@h%@5I?dzrllV;?j&dUzE6K{u65%VEf$Y-VQGo#3Bsk;V&BgaAB0*V=Y`gl_ON_DS=lldevHN_Ao+|;r}uA z)=^dMUE8oA=%#dUI;2BFKmoa3=!FZs~m zQn$}#O)j#HG^C1NC+v7n9qVN7sTxxj$rm0Q6Ae~nZ`}GU_fgha*~35IoUK01&aO7f z`f9Vs6EV&kOLMv4w=8nqPURRa7iDjIU7d@)^^~H9QA}FuR+Rdi`|rLCXnj1&f?bRf zaj~xPRMm$Zc1RKiXYu9bB2Q)aZkuV(H`MVss2-lH2IE2XIO> z(L&VSPTabBr$HNu7uq%D612A_Xv-+Af2~fP>6`wM+FekcDcW4q&}RQ=s8?y%m;Sc2 zjmgn_k~dyNSozIRdFrthT6(_uE#ck9!5K5|w-SEi#Z?>;1@-u)-YeSSZ9i$M>D>AA z;xFEeT+=>wdZkHAqp-M6Iaf(teMzvX?oY)1wZ&Dw-``dj`h4f#?9yEun|Y_~n!@9* ztRWU=aKw(rOL#_vZe7{o!)oqt&$!}}IAPR_<2Dg5s5 zD8;Z*r2c(kWx}VQH_}>f%pN6RLbZjgU_rsLpS%tJUh{lpBzPyR=7H`<=!37;_E#Qr z9ANH@2&#r?c6ULrQ*}`VntkLyewjFWi%nKJu%*G}R&?cd^}X@H@a2NnTyOu>7nff^ zF^>Pbx5a&wk&h(e-IROImwMs+VTS@W{l2>bjU0(7%o}}!S~B5wZCn}<;&r~A*LYER zzh5WsGEq?eE!*?w556BT?w6kmvY-?f*Xm!mFXZsPlkSr?D_=u(89F{QRZEiq9HPRRgdIV1 z6Wb{r-Y2P_q2OGdxSMpCM1f;~hWkw-^b*{91k-OWEgKI+mh@D%Fls? zEPc=zR`#HOC(ZNEF=n}QkV(E9mP~w6q3L*NeE_UuN#5G%agAC7&t&kWGpDd&!Gp?b z4oZ zbRL!$hsCCPGg=0^8Y_CZWudwKtV%eV8w2v)`V0m}`^8{m`nm*^$lM`&zT!MG6*HhU z`rN1uRf5VY5WzcX5fl!y$7A+=I&v!ek-d>fNJ7Ehv>|&VW~2FcZ#=_)? z7C&yGe*VzWJSgigeH^LuRp((*QRI!$vhsPUiqnU$w)Yy;qZaAJnR0fJvcF@wi-FEm7JXPk}fYM2{fOBFMZss4TQx_47nyMdDHabWIz zp5|$#e}cL@lx2NUYWb$U-tOZaEF&r*CDJ;PI6{=gR8v($BWGcOIdjZLFT1oIxAfMTj?!NI5G)xqxTF6u+l&D!cV&H7;r%;& z>#b`(W3wJcgI-?iW5MnN?Q?-7B|=YfeU#{A2PyCKV~Pi~1iK*oRufYG{!(5FG9vM+ z?YR?=E}&n(V^O^N#i}^0#ho3Z4~!i#vSy|7u$UuL?Zcd?;Rv`7*H(DeiS|&*bTslh z>9>6{-5 zhxGQqSjy*&oD3O$a|aCcrEYh<%gXNa;-!WwNv<4&p21$Dzr4|%y&!vc9iN`g z>Y>AHuC93pKdp3Go$tvCW=JNIdE*a{i&%~}{+a>U&{{^;{aD7M=9W0JNOZ)0R;Div zN@@0ijtMaVBAK-3?XD)qnyX*B3iP;D^z7`ipGzHFY*l^;l&s`#buzt3L-H%O6n0NB zEVGS-Pk)RWsa^5sM=!{+hn;9%QRS`NqFOO)#L8SSe}kn@z93-3>JS9PPUsgDu-1zO zv!4vix5iD4Q%UT#16aQBZh z9o2|?;ZLb>R(wQLZ2H)ILh`-&It}gB7gp6=+!e}0Xh_+0_wmv3g>z(3Q$PEWij7IOSmxIH&`_zBtYS!z4HgkaKumYG=C?1u!c zm*zqVN@VW`w0u6lY|&)m#h`PshvBau(41#@X;iR0xt9)A1RB4V_LBO8TV*&wIXq;2 z_)`Z>ycAJn8W8%}#0}6fZC}qpOvhiZ$X}NPl{Q+5!Q=jYhV%0+y zK|qnJM-3{k99h*3=4uK zFOMp)IlkxN6iNXOlW26weOz#6nfr0oEqfqALS5%gY4@vA8Uxh8;CtPR2C}k0`|1-0 zj~h95f%s&&#|FY|I^JwvpMGQO8#H|_18aSvAMcwHv>M_+)a3mKCu{0sp45E51_giFa`R}O>~`_^=ed{8+rY+Vgp>iBDQ_rTPo`iU+Scd@ep}w8H=Mkyj1eHMPKbI?i>8 z=X#e7p0X_}#OSufYVUvj|2rJ;F0m|kWU-Xra$`Peb0DwaXQaqS8?9lof%otAO|LV_ zRzKt!{e6o+8=f+oBz7-iQoa%l`5^& z;5F$xdVq}4=dSx}%jJFkB$R%T4ujRkcc6H4=ScR~?GRk!WJSzXup{{JkoD_yu|L1o z-5mQ?xw>8|wRh1^K(_ONkEJS=uaRLtRpO*dJ^zEv%a?9?3v05;k`y}^cWALz*!12` zxIZE4(E661dpPdBka17(eIdM71CC6#m)z@WbH3YMI}#NKTKAHNpC3`&_tRpON_fRC zi65jhJ|wJO$|&aa358)TgNSLvRLVq*>lhbag5D0_=c)Mbc?aVY&h-U6DC#i0(HFg& zjJ9kVZ}&dq3N=usd|CnT-cL>{N^RQ`8GU_{zmq&dOG9(gV6wxV zQG)F1%}F!H{qu&Vgm-AxZDoY>rV1Fymdyv=sjN+N&!h6e53lr9%v#(T-#$@dy3Koe zgIxhi|JKKI!+qZm537Ra#QSS*>-?y>sAL4Lu#M1nmm0&`ERcF>m6kWgMFF~8$jF8B zH|Y4*@(U<;`25UwKXteEwedI9`woDV@o_11JR zbwhskRB^HMhUwZ_Lp^Hn62GZ=4Jj<1osY?Nn_0<-|7(|WrC4e1jD2#F`kCfoh|{+} z$Xm^NYySCpqQCX3PWf;pECk@~zx>Zk;Bf4M#F9O|kH*msOtH=^$@jDhf;H zDTur_$0l&4gd6NiIT6++raDKbepB-D@#!o3%SS~=M=Oil-n*#~5Fqn@z{ue0RdGw- zEne?mWrrhTboko{T|~#JDGt_zKkG0WW4@->qK?Fo7*{Tm=%bNK$_rz9$~D)y9eyMf zC8Q?W&m9A(k3YKqHcap~pDE=pCL(VWGArA)XV-A@aGraYa~vwamjyiN+j0-WZGDqs z)T7q>S~xn&JX&`3u$GL^Im@6oU*pV~1>sh)@q0DR7p2C^@0e_U3U>BZ!Jf1hk1`05 zu_;$3X_T~Ix_na!Rdc9~5Z9Pqw)YH^%mX3?kC)GXg>OGVVhFU-)4vM$z1wV8Wa*WbUEweoVbEcg*U*C49PmpAf++*6dc*loTV%oXcm!Go|` zteth&trgIt&bfM(UtTKJ&_LjKbWhxT_V@}DZY2oEO3^)_MUnU<8z7@mTDCO@(akLf z1EZDYNhgxgm5(p4$EQ~fj?Lu>FHWSqL&ekNdd+xh-zQ}rUkWF8| z^K1_h^y(b2!>w&~2+}v~eraDD|0rr;hvmx4&o%J!DmT)sN>A6)(lVSfQ4tn?=iO^$ z-(NA6#yBW@^L$Jk%gmZu&H2Onm$kkZrJBSfNd=TA-Hq0M6h*zGFXgG|>Nc7^s-J*T zj+w4831JEzMw=O1%!_N?w^F%v9^VtiB}^q!{B#)iXM$1>)GStaa^#V#c?Og~1;^&R zU6|m^#lwy8%^fR4Vz_7|+az|&v}ycucb($XKi`BG5=RpC@AZSZdQE&`thEi|aQ*WJ ztVYJ)vZlJE?{~fUR6%e#kp@CSn};1l!9Z-oV-&+k!^T64tfHL*go@vV;XEB9mcUYR zo06!hYi4&!Lk{Qkq0b*7EoN~-MrhgO>AISvvVo*x461XRanFW#afQ4Qu>Z~Sf|LmD zAp-XxeKFwcF?d^P*nX3C;|t{#kS*QWS!~2X8lS@NlE>W zlZT3*6zBJMZ-IDsB$<|lfu&3qhmDzj#|*Jivbx$+{wyh3m1TUwHtOu{=zMp<@JY=^1vn7EF*~$>W z*N4xXRj;g`d3R@iO7xF0ZSB%n6?}bAv*;4hQDBonq#N6*1t*5PZ6&K0&LF;VOC-Nq>u#bLCM`lf*f@lGGf0;^A@d>)UZJx5ZnIe0M2HMg(5h2Xm{nvw)=aA zg!Fq#WYCn*eX0Rd+GeJ)H1h10*=R~pcHLd=q(GkZofq>G`8Jmf-MGx!P&aAj2ehuA zFR!B?dT^TN@72t57EhtIncSL^7MtKRe!zuvit~6lqiwPCNv3_z zBT8JI)-nR?c7Vtp-LyK%bmNr>7}&>JkkNa-ZGU+YxNpbL=DRt&4V5YClCA}5MB4&# zBJofIdhxSsvQvPotMKzmDTC{#yq~|xC*)^}>ERUDmRRiO0i4em8wUzCiwG?Ck;&HiCs;iuM&3s|3BBr6ucg zp9=lU*LkVljosnXmVdyvvtY6y-xMHbBR(YM{6sezV=-ns`So>f!GrMeRMVnVprjo? z^ci(_v?SNgZydjq^}@>Jk&F4mSdjPJFy)S2HBG9`YgT$5U``x`_L2pvwBsXWnvmBv zfT*IAHYN#TnFA!*83!^p^p?38QOE%x5hK3EF^V!YTFQFP9(10)zUqNwpt)wysXqR* z0ueI-$90{Y1Yfq{8Wf(HzD3|xR#vFlm>Ia#cqivY$FEtxy=N+N=95-#>0JqS&VPE! zAckL9Ad5FTlio3=q9~@qhjlFqiP`Vk;ZOLNFz0;`#hyP6I^5TbTTWLZF=iMp+2hs& z5#Qg36=@NPwLCsf_T{gHUE@Ha!5I$ZTuxKMxs++`9_9rw@*EN4R_>gB68Juk0Q;8)EKod+*^`t)f%6?>V}-(0huj8EV0VAsCI zW~88!?uTLx{6+L&BwGGFLWidRg|}^YcM$2tcVD$-$7k_CI5IrtRl?MVp`NCvWT*i+ z_fO&_C36qB&w4Pg=#-!`9(c7au(r`vGLn0t8hD9BgWhFslf4CoIBIorKcWMkS_2%2 z!*9E5SINA+z59+%xn@&M(e`dWjV{dfV|hO$rJot&t*)ucz!YoT8 zCT{gmYHeh@Gq=PqU)gM2X;JE@lm>kzQXyprB?5ln1cod=Zbnwt*| zC30Xq7y<8{efwcp1eW_ExW8qSp^G13Bj`as(t_;@E3zUR7?RX|{pDN5&vw>|`7P?v z%FBoEe=E+dP4QRP(8+@aWl_3%nMae4e}rb6eBW!=tdqD9BIYxn*v zfM8$yQQOq-=tY_DP!^%)=4D_WzyiWM-5|?@dG@)zA&T3M!-VFF-O%e2fuC$TSxzVP zZRSe1L}yWRGiJyuwxGss45U0ebEhscJcwB0w1nxO zvDbUui41?ILigNyt#X}Ms!huF>xj)~}Z%&6FrF?gBqxniSX4C=IjqMaXO&|7w)n$P% zCmY`gy_V|_LHvuK9<0jIBuHmzw|~0bNQ?dQwm!yd^+%XzJfne?rOIWa=Msg%hv;IC zY1WS)dxJl;=R*z@M`|RlW zID7CA!~6RTsm#0>cpG}2AGhppo%Xr+;FhQu_9?1&*hEKWgD=^CvqDlct8+dZ{U9_j*RjuskP{yGWw^`%xO>mo6ys1cHhPMB9LE6LfLH+Wyjf~ zio;s~O;qI8vKN)u=)k9!7g(1GAsg}?Xmr`Gy+p`Ac*2D7I`QP>aY8KWDdUGFU$kd# z%dgg(J+(8P1Ohb>5q;(wNZ3;X^TsGXh}PokP4#N}y)j|4??-JUgzwE3g)?G)&vKvG z`>Mj@A#b4LEQWHgIt$GDb$4>*eV&z*7d|-GPpn1@(W;AH_2&?M-(ctR(0W49p{7bqDZhwK!sK=pV;XR(o=;D&uL8^9Hjp7ybgu(a}Q> zD9M@SD`eDY?l3v!eGkYi6O)i9y2yS zN{z2o12vJgp5s|M_Yz%t3Su&@ctX$g=^6wQ`skkN6Z7Q3U6?Gh8nPE_F)CLch)og) zB<$|?zduskDovSa&JwJ5Tf+{S@g@AV%b@<)!^7KqusO{@N02x8H0!58K@4+qn)iJa z)misy9HAmDFcV9txU8cjaQZ&o-CkzoGtZu8+(TkVxZyKzojo>&eC7%GOx8tz7X&C{ zkoiD4TGr-Hl1$XSA-Eo#~hBs(b1M;G^a5go&y#YKr%PxxN7>_E+#D;&9W zh14839dJw)OvMpz=}I6vrd1Jl+V^c(0_^)sBwd%(YU4JHDnw!WV6M@c zw+0U|$n*$<9ERKZ4vHYMk;f^u0tsA5{9E=A^s@+#bPZ+1RH`u1?~7ZSKGW`pm7v4Q z*y#E$ehn1gHm<-;JpLwc!Y)nhzxVQq@bY@_i{r^HE`}~LRvX-k6m?V1;o(ZIbsT*C zdu_hT-0`w)M3H-3OqZ9K^;B&6Ly+mBL8faNz9-W?K=KEJJZq1d_@AG}Oq{VyLC;bh z1QP;gWOhY@AJB71&oEfyq))Z18sg3`|d71{Mb^CpxO6H1>krG_}`+ zaPmApt`bLe0f6GD0vI}8zXFo;PqIa01(E%x=h2=$w+XMcNQfTTxl`)sk9+h6 zGeW-H%(Ox9*fYRm9+kUg0Y1b#ZK$&p(Tu0^?rpwZUv$hzQkYmsu?cfHI!l+L;{GPV zWA9GaAU1*Bx+Ua<*E>6s-M0uKY?@%B$Rj5gSwopIfx-aB-#by|$ccKqXmbWm zR5+L31kVX7oxb}Ia)CU zdQ6z3bIz$Rq9zUmPCo%&#%an*4KotLE7X1OfH3GPW~U48Lu7_M_))F9yhk?dht=5 zFtbI#4o!|i_G-ocy+edTJo)JJ-}Y9CjhBQYZQ4+>0kc8v1y}U2>iO0Mo;)N{&y-LZ z{o88`J;!ZOmfg9u!6q3*R;x z=$l@kCq3s&9i#e?x~OT<@6mkI%&p?@AEIkJC%3Pk7!XJfd|%*k|1qXv8eu;RnX+~& zu{!%U+Xlf(CSHt(k*^;FNlN|mKcvoZ*kK3k@Z#ICmWzNB#VdsHk-4)%J&!f{GKa>_3EuI6@=*Wne&I9gqf=4@8(Zz$j!- zsg$0SMh?!#d8nyzOfTj~>f<1a=`K31-V(!4<xNI5OXJyGc{wciJ zj?2I)eqq+!Pq(27@VkjIEsJ}<`zZYGvDQ1oq_8XX2w8y~x>aOt?3sk^@D(PfeGDwG zmod)lX~URq-#y8km!-X(n1SR%nLR#k9#P(B>9`oX?I;n45~83B#1kE@h&elgY;~x1`-5h#~@VQ0Ue@m&Ap*Btni2 zAM9fe0&*(0%C55_XQUNau4yCVrxuF5yu5~H<$1R_nx8u6(A8PK-Fh}!$lr8T1Nv{$ zu+uK6vt9>yf~8+P_K&BiJ7m^h(l{AEs`$yXjps$-+5{hd@De7Ti~a)8=N^H(C{)?=Qm z%VuW6ko~Wpp%Bvq0rO?J!U=j2xR@StjK!OWdCi}V5_GG)rZSXqMBT(g=U%YsA$PbY zM*+b_Jq8R|9gx1=P)NyFz<-zCJE9?u2sg^CzJDe3v<`G{6<#|UvnR>;l4j*YE1V5h95!{RSd(95y&wY|QblQmQj_O_zwc@a-C^_G_R zaa!$b@l#wWI8@0$(zIgs6Qa=0hq2|>9-shr@>*kB-%>Xd+e?ve+6OX)%U;|%UnT!S ze)+u05phJduJ8-4KZ@P?@g*c68rWvocdjWO0}j$F-!TitAzgCX- z%qqPobXh#w;us{;$*(`3^r@&f3Jn=PLIh#F?%dDwpL344p;}R5M9A_FXNF9O7Wf5M z*FE1od#&?`XGZ$We9zbZrS7dB-PXr@pKS$?Zh6D<$TSfFH&cM((23vV3r?wPb*~Ak znsEB`x8yx4EUd6#Dx4h9YE@=en{u0cVve^7fBTrCd9@71Ns57*NwTAM_#a|el|5J^ zLiEB~`@@gH8vC(WnfB|IImWG3$Kk_URj7sbu<9K0;jHFwT}1ajoCzp*VJ5lKqqmPG z;DCRzjlK)9$vrDE>|NvAeKv|W&G(Zxn?4pwpS4%~`IQJs!;=Fey?oi6G5ME&j8nFu zes1vk@71neBpcrLB2TP}l^v}dKhyP0c!If7Yh?R6yJVaR2rz+eh3E)}`e5f|4&+4E zQkMrzc}t1Bbp(}EtClZ1lS2J3$RIV+Ycdz65b?4E>a^}bL47f4rBU5WEPe!Bu#mwi zM}zZ@&H<1u2P?iJv5DB^kk-GCKO#RZIVRWj?A;1z_i1r?j zWGm;GE|Lc99rJdmz;Lky7C4uY)3;D10TDN*7cOTA>M?tHLuJ*sBPKD0kYWrk-|9)L zNn~yT$AXrl+|#+(qf56w(~7vKd&Z`hYoys1_(@A_Rcx)wNerA6Le3u^&J^MMk5^8E zKto|CeB&VsiAbvz@b#dtaWW4Yyw?M51SGUpujnk5Dbn+-XJn+Yh7e(EihosTur{3fUoPQij?1n7+88)D}*x z!y=YMn$s-ck|oscg%%eY{VP$RuvK&9EK=umf!0psaDz6>SUNm=xgK- zl^%s>ouU2LdeI5nk&LwvD=sw|B2oe--sWvU$tOI%C09mT4{@veMG-f2sOfR(hPwLv zu#~k=ualqn-IC~ki}b4~DajJ!V-zO?uqbFLX=FXCG~Nt4#|VTq0RskN^YIcy^av>3G6 zwS0F!UiG;}I`_IoabLAS84cvI=2eQn)kwO$`wLr~L9+IY7`g`u`3d;DBAe>}y7d&Y zBD#C{l+Ir|6)_mQw*VNQW?LKdnPxr1>2uE%I?$4vsvMF-t9{&DnvQVJ%m8 zrVJ5ZzsLv0sY%T1?sw}UX1^yOd2JyxbTkDj3J~Z82FHt+DcQ~52ApoD5BmDU= za`Pp@136hSY5ITD5~wJ|8F^f187OOd$j=KOoflh1FhxGPjq2o{)bLX}TG4+m?*My^kRXR@4*=!@yEN=eZKE7_x#e?7^Fb_btkN|HW?6<^=}Id)H_~}uXg)hyBpcdvsa>a( zKEwyFmxS^E#b>iID;v0ney>5))N;hCjjcryIVP8%^-D|yww^1iu3A$JT@WfXpLkXv z8rn4EudGA{(97^8K{a|$GS!@+xes_CGqZn@-v{xmGku3SqrK0EtTfCky(6ZIE1=2V zXtgIdD~wCf<8= zZOVbPvKXlbV6ouRZ+Xj4MvwY&C@;WrD3#yVFmAe@3bdFiptT<5EYIlL9n*xDaPt12}*_UHG8 zky1FoOx{IL;>OJdA+u@^&iWsnLKQG9z4l|Q%v@fgRrce@Cwg}W40Q5hV~^A>H*CQe zMWun!QT5r(V7Ygh@|nZ-$20u6EdA&6py}CBJxJtE_w4nmuJqJxKejEpUn?9+W7ps7s= zej5|M^SvtUblX$cXIYtHwj>qPDZJZ@Z<5jrg*o^_&*{*^*_p+Z=|a`Z}%jmGc-9(HI=L4?!==@FFL;Nh5tJ%af}- z{(qL;4+&$xH)%b&=5tGEZQ%a{KO^FIUI0fPgQnTTBHM_q@0?Gpf>!k|F1epRU8?;x z6Rm`6J)RkeO;1m!Wns~P!QPml)pt@{qqbY?-?&bNZpE<4rd(T|)Uv+u`?7BepLXLK zO!VzzvQJZlHo@LQmcG5v2EQ$^0S1FVsSX=_VyOXxeCyC6$gFeeucfUHkWS|*F@#~Q}ye2%1=lo7mu}Hps%81OIm7YzzibusKCaS> z(x7e05noO?10RX;C88`vGTjTY1b;GQ}@@Yk6^;0@3a?O(uX{qlcPs6pDKV5FEMEZ>O&&n?;#lVR@0A}PSCW7 z4o#)ymisk*(0{pp12=H6svPk@T@?#dIO+37KaH=PiQ&QD`)}3dJ_( zYHDf>jf{8~v^>(Ipa*QINaG)`caY7~F~tA$M0 zr-MrP!T^DD@#quS+|QpjPDZ~(8NX&;k)kaf{3y1zpKITDYH2sFKN3`ij#<$OYH8(> zq=c>I#>|YsA8MmDR}0YG$iUbeH@=p&tNsB#mxgTbu2*}B0$ZdTUAFXsvC6wC4 ze}GZm`OzY^sslM^zE$+kJq$mRZSG!ggfqBwS=@!HIjfIZIP__wvz(%?OWY%D4cR(qiFA*hd!o(crfk8IwK7C8Rk%D zaSt&^CNZe8(E?w}rIjKP2`w>|Zn63EU%qq+t)iRe(L$DOMIsgJg!Dis#4(E5;nB2C zPENRVG6Ci9j2Xv;g|%XSe{&BOS~w3=)X0b~!H|}j>m=)Iz&xhTL#WGR`TJnP7et9@AtXX zh47y$gAR+e#GDWLr=@ytUf?wGAmOjZeM7@&& zP@Q&!j`&aJRW}%L5^!f}Cu1Wm`*wSoAJh#X+lM|2R+y`uO~@Q667vaGLb&HcuzmE; zoBm0?y7qIc&q0NN_FzD_0*3Pg0pG)o?XU#vyKwzqQ6U6f(5P;nqbZ`mlt8$B`{m30 zBW{A0oXUl|SoYtUt1?T~=bPfP@l~_+w4A?WHa2KaD*UZ^sssCvPn5dwOaw6u4bk)v zp&#+qTZXW_7hr_y1Z@cy^nv)fLA@7sLy&@i6Ed`z(otdyzZ7{(*{feP^*C7GokN`O zMaPZgitcLy>K)5%YUIBAbQS2#{GK4P`riv@Z-o9D|)O&Z6fw;h(My z^vNpM(^j95Sp3wxHX~zwZ$u#Z3@c4l1M6j4O&)x~GoZZ#1AG4xhKLDMp0hpiPuN-c z!>K{!S>QJ^Jcfr-zf z^JPaZnvGhNwT@J zC+?MkZna4bw(QL7vEfIUy^jMm&G+I*#!ntXh7on}PbBrRqKAT%-z~tld&WCk>Ey=< zij?SDr4GH@$J9{Gu5`4tys&b=7~v?9@>p*s3JI>1P*&oy$20r9egv_F*f<-2br070 zha4q>hfy}Iid8&$5)1mu^A<9kqM~;oS}F|Q-T5N=>Ef&B&O`btDu)qZdt>O_&aaP- zU1?_}oChDXZA-cgD*#^M3jPzX!lqU=?nN;HBhrxx>b;Hy>0hpD%LavqHm@V&mLz)9`6NyxMhEWvy3osdXzD z2DKDfH{Pi5{HTn;(15kytC_o+$f%16k5;O+#Ku1nPa1dpH+Qy{!$-nCh{3mTdykU{ z_F@Q89^++AHvvFqC&A(5zjj13S`b;-3EN6WPUCf6^-_+enWj{B7hQUiiWg*j(7N_u z49)=xCj@j;1i7J#1on{);?Mx{f8tO$2etzL_My=#!w<2K z?7P2IMql`f%KrXz@5Yctl^ZJ+p}aZR7e zz%Kl2osX|>=fDzW$o>9Avk6No>QevvOKPP2`{hTLo-gUVVJsMhxmD0lBU|>dR)$~4 zh?+7V$bF0vcYxtW6&@e3E8BB9ck*8>)kc?>m(RgaoMo}+_ESSv0k@$K^zlvbZ316r z3L{#N%3BbK{dc++y(CDk27bP8>`%ZPX7l4%xxDZ;Sv*1*LIe7|{CsjfC@le)#Hk;+ zIbXnKmt9<}j#KcsNEEJgGkAN^8v6fs)6vkJubQ}Key6@nC+WDjT|PjKCr?&wF4#lW zPQ3Tvxt+Zodi*S80(R*C3eom*7h9Z$c6iBCVWMx%nJrrQ#g!sqM;vpwd3fo0u*tP* z%He)O*(EA5-)T9|uo^&))!052@?EdNl{6Og?E8$S1F+YJrhhYSw$bPiC_I1HLlrlU z?u1oc(&qX%A(8Ii20I0rH>43f>OE|GtJnR;DvRfQx8s8|mgtR-I*Zmm?~P{erdwYO z4VRxlt}caM$S&qa6VYKb1;wA2jxkT_*Ug+99kXDCLX=~V&G)M}q&31Jj`-CyA8mrR z)eD#Z-CdOT>+W6Bh>Le*9}GS=J>P!pY78OU1_pY`m=kzLzlZq+1114 zbrfX?p?Db6BtE9t_38o@!tKO%yC6~dtJKMMZuatMVOEn5OV*IJ3KGy(=ivetDF0CtsU!+;zYAefo;0{vAj+-2t z>7b_PP5KDa{@j^QJ0+k4p_zpp<I8sZQ+qwSjHhd`8X)RIpNlX`BP3ed4LFD64@{eO<_p0otp?7gT*v8@dRE zO~)*$yHrt*ITujLe;MLu|DF0-kw0=?VL$cr6`}q8>N*`G&%8T2#4{~TJzkAI#ZY~P z6Y1tkAAIj8-!E*s=<@GVYljSCllXZGI}VS=^=JL^@Ak<3TaW)@(MFb#%x^p$2! zR~t;o0=pnbT~vPK*N4LxVPrT~wsz9&ua%EVB}Xe08?S!Ej--*RdmZU^T&X9AeGh1= zT@0{t%esK~2_ky~-smerQr2%rPb4G#%cM1ct$c#f`%mp^0p}_8Fd4{8;7<7%1O`$Y zNmV;mJt}K&2*BZUkMQ~iDsL=KwxQ~|+A{Jbi(MA-mb@b}%viium3ux37zjv->e%Kk zi~NU91<3T3Ri7ct<9(v_Yke9LF2g4`5!5bH^P6Tf%`7s9uiENhFgCd7q2f&c_OZ`R zFO%R0=YnIBR3MPb0&w4gSK7V|88NTYA%Y_0z$jB=qhtpBF_snR3|2)3wZ}ENLj{ zQYBrfv(b&OZcK;0A5UZ{xM;^2!EKnGr5HmvBx*8r2=Y!Pi%cealBq`Uz@gCk^J>q4~h8rCnsayN^G|-0qtcptNU_h{#hcrUmD!uvq z?P>6~bLYEvbiIcyT5W)F$gXUSV6fuXC-qcxYWVfV|8QDZT)=7p0I96`!&1TC3EM(W0MaCF@D=Whm^bz z4>f%sf>?uVK@3K3#zaL$;cinVKT+?Vy4=ZY^#)=&t|^PzP;Tx^7spakQWtEQ@=&Cs(s61pP*6jG>8j-*kVWgZ6}>p_%q1xy}~% zR~^Di465dPRbw{Po@k7EtM2q!tkhm%D0mPMFWJ_%p5GNKxndtnNRqf*ycznVAJRAi zjACSn9UF!m0^$3wL<-)k(j0EvTN?%UbmUAe{1D?m6SmW^wUqO7aVVNV)}S!Uj0#Nd z%G?Z|%evKeMud5LCR&vILkNXQKokb3qP%KvkOW2a&6YYKbc}F#w6arxU$o^^?O@rq8ZK&e6o0MczCl z#>HG`1b=fbr|{#jicb5af6ki8mgQ2%dVkZTKGqi`w(y3aBY*hGuBp zbx+h6@~uiv+o|R%p-QMM`g0oW?iGHsg;B3!*zKUmy*$da3H-j6*-fB{U<5J)6}y}T zZSEJH`BmQ?k{v+K%B-xchEVS-viZAQPDDhTv3>j3@-eNrq@==29#CZjG@!A?p8)4H z|Bh9z;KjibyBiG&RT_}z<+#%Pbxndk{7hJwvm?%+Rmn@@I`_HC0jVkj(;#u9fhyh0 z=Y?vR&5Mf5?sqcR-gr7Wrv3b#>p;)A_U#hZimH-uvr*~7=Q;OAe?NM5Pg-VTV17y6 zn2#khFUblgCRWcc!gq7V?jpaO$W7`QJAoqK6TI#1`et^=e$KGpKK73mK<%s6_T7(# z!RqIzRJ9(yP4Mefk1W_JT1iNMpm4B{T1o!zfRd{aVx7FjeWeU3T3wJkk_a#}d0*?z zzNchr3ky}J;34?YocEJ`|2t@8+S$_Tj(MOHJ!EG#$FYUnoPM^}Si+?iA8sH!3`w7L z%l((xi~lJ;bYglGW5J(nKqPEgYnZF<573v2Jk6rWDm?B}+&0pXJQ(Z{Nsgr02i*u`tK@#Pjti zOk;c7EB&WQNlVLrp8d(Ts>e@{Y)`auaB5diSaOjDFMs{I@_wY3^TsU5+`rK-FXfu} z9@bNN@v9zss-~Ek3EU)X=JEA4t>KtQr6QFt^Y-s-8}L_IpZa_yU)SwQIXU@08HwiL zf<)yZj(=npblPHAv%HkrqNUk4{;r|UC5)W2{WQE7RWuY}P-B>nVQjA0ep6WVUp0YL zaEWYypn6f!g4Ub_Dir;m_T_C^B z8mIhmU86wkVBcH_kt5d+;{Cqk(+qG~JedEi&YX( zJcPpB82y4j8ahzudg&>M(n85EJrnMW1!)PD%M`H~7vrsq`_iirIryDxD_CfM#REq# zKKE(mH8msdlC9wNE|PP_KQ?!#F7ds-G_^s3B5L94;ClG*E58|CNmTVi;mv10MU~g| z%!S-a)SXM7`+TrpG1ePrz!#t3;&KZqKi|@DIrVD*-?Q!AS?#4ySCT$Yn%~`JqrH-xo*|dJGUz>j1mFZ46M<0q5>Dst7^x+ci!{L&LFuh_eji<%LDmx<3 zQc-m$Stw8XTx%J*~K9_KCK>y@?PNk?QL1*&DU0_ zg-6#z*#x^x*Ts`Wd`%=tAt#7WD0j59-A)P-M3pjTMY)#BC_ZOtkm(KSK00LkX64Y= zM-EurC6XDM8QZAXh_hz53WrS`F4_%zu8yh`W1>8H&0C@WJ(;WV;Zd_Oo98BCf}%_e zw0U;Qp3;0rHR+s;O2v=|wS>YuekgrYryPv75yQ&8+@a^j>BuSx>b_7=QuCw6;VZOU zeTbkt-;c(zGTS>=JNd6&lI@C=i)+*!N_ePSJpJgoFA>jPf$xSlpOzt_hv9x5+$TdD zxSH&6*74Wm{tsDi9TjC4wGRspG9%#(-7z#sN=e7i9RkuN-6bF`;Ls^0NT;Nvlz^n5 zfP~UzfC7Sae`lWe{e7{%zjU!!!#(Fd``XvOc8$lJ`vnCw5jqo2ufDQ5nh<%4WMRQF zFr!iD_e96W5$kd_oGFIsN>VN?6bdyl6*bZ(B-9M6ep%}*5;PS<5|iU=4)qAu%CAvB z+5H4g#ux_9V>Zv~{j_2jDC`t3BIB1k6kXI6NrC?}bHq{AnjKjYyvMB)$-uAD;Iv<};Wl{TlYaLAdY;sJSupf-4l!c}7xM5B$UoFws9Qq!#ZOJ9)Kub0N@84B(+CEs`Xd4?BDQ$f8uX5nlD8gSs*g@v zfheuC6c=@4^b_QBCbn}sCVw(;z}T8$1L!cOpD?*d%xw@6n6U2uH&pu~)tIpFsxJ~N zOR5_#{?Vzvc-pr=EOlDrjqq*?!9W_r`gkDGH3^fl;Kbtvk{CFfGu0$z%4!Bd9O7*f zp@=Q!p7CM5gY@52OJ44M=t98^tsyb+=@%%dJu-SR;G%=pV)^)laPRV(x#!nhzF=O` z(Wm{u&W?_|`7{$74jOo0rc$jYSY(Lv8A?p8^>DFNcWm9971Zsye*b>OC;9-989&_j z$i~ZKxj@iYbj19>YuU-rV;xC8(WYiZ1TM>U3$AfI{Et{V-5bR@PswyN_O!KUB?}*!SJriFdDiRH{jB~66sra3Z$jCH?KRhY+Du}_qLat$je|#>0#X7+YF^bAsdo> zrUy?UW?ADA>o)|LPmbj1fW*wPxQVR1Le2qO4VCs(!&vSuVr`k88D5cfI0+78Ec~#` zk{szk5ESrcHOxbAh8TREd~SiP`=^#qSu)(k6EvR83@d9z^9v(#`k8rzcj)`C#O{&~ zv@{9JFDysQx3iA1j|{HI;Iw(E^H?{fT>Ylj`i%J_dnj=EwIWY_SuL5sOSw#2)#8t^ zTWYhClv=1a_vp#t(>`U&pT+hHewd#c#mc~_W^V_ zoUxz4nqO{G=f~jxzLLl=L6F0cCS#I7tbfgxSpi4YC$dbxTidU;lGhh~`_b@6WmzQ` zA|DP8^t3||b=I>t8DRr1iuN{4*5WyZb*}x}h=-?V-{nmCSq+<|8g0PmMv8#f-uLV# z;OHN)U&j2GV0*NuomNp~>Lz5%DM)1rRn-5NYq7|*hR1DhmKuC!8NN5=e$?wWRYNB}EN$KEjwNBVUuV1>4e;xY0 zUstEKUcB@}Sv_9KFz9a4<1lSP0Z3pnpOuRXO5cRZ*m&OKEGua7;gefsB%TQ|@Neek z*u+pu0KJi82LI%MkpX}SyUn7Art9q7W}y)e{0=H4{z>{Qf6*(!KJp@~VMs`XD;Xe_ zwdu7Y(7DqZ>x*6kP$fjf7yg0%@{az^dk8t2+ySVC7T1aW|JZCMgq>gtC{vO|*8F`; zNVguyc`gPSlE1e2t_{>_S)R?NIq(}5XsF<%>K4W%W{BKsy&Hi2vzDje9ah=k9p*ul z^?nJ~=~;UCF#6X6A~3KN&la1L4P1hcBIqSO0&vdX-4zPJO6L$rfSlGIwXri!IgOsk z$jRYY#%m;LKV6Y1L(P-5dpdn-3`nK)J+R%i9NKFg$y=Uw0c-^0eO$1a6_uUkMg8ze z86#!~GF#s!;fUa4gOgoy|3d!4c|Vgkac!#WYw=D+;=Gp40fh$06Me!KDPxT4v3R@|aVRIYoGo()L=TPV;cgJ@){mz!o)Kh<@>vw(r$~CIhcDwVBG+ z#BZP8#+YqKZ3JLd$o6-Hq(5fb+}368zw=;^w(ZBAq5Ae1d{{&rFpJ)1i*45%1csVi zfYr#eP5j!Ncaw*%tJ&57_f*vtj+vIMG0SOpT-{Uo7O+_YhskE>NX*z+{gy$)(+O4W zT^lcC<@#${uhg$)Qzuggpp?mNFmTT547o5-MWx6bxa_mh%+|GlzJVNeu(~`->sFao zej?=qyHmkvY4H~a0m~LM5xCNlXszQ%NG1{LpXvB0r{?SaQ5$}&Ovrs!V!?#wikj-d zfnu-y(@raj?6l$j|AInM8IO*B@nWROabBUzKsT77yKZ z;0Qd^XH1I2W*vg6eGT2wjkMEW(=Q~6B7SPNrLCeG;ir8L%A%91NLu0s}pu7t1OzwxrzgbJyUp)BdHM{FJgv~8Cv6yHL zC0%rzq$j!Se!*o%jDhUT|AB0H3#5M&txbY-(EG6ePnhK10~q!p9xmmT38%osdld?X zgGWIgLy13MDjJL4C0$=s%eW8wMix00Q0wr!-+eZH{drz_N=oK!HRFuE5IY-eCkD%A zW5nY+C07N~S{jqE_vT_W0VBCQ>IBa|Nb!YRf_H$b|I2W(Vu&$Qt@gTp=-NpG>^5I; z5TjrEjK7(d90^lDDQApnGI8nTO7r~^@vrpA$c+&MScX<@n?EJoK*4wuJi-l(6RLxK zv|~2WN};_l%W_#+F1@2Tjkb82H8E|4deaN~ARdUt+xbv^+7x&^@ua24wx!3V;cn>W zdarf!F_M%L!DyQ3h;$8Y%27s5(YrGH=uq`5I>PB|WaZ+?MjCXK<7SXdQdHUdlge>% za>hYTdqP(us`0YMrat+9BPlRem0$b|wgmTJl|pA;hLg~AE|FAf%l>8gr$eH0x#cd3 zMzF?_zQ5kQ_4;O=&kN7nQuC8sq~gZTtcNr!TpQbei@|4E0FS zW0}H(Rdt4b;FJ;T)MKLu)ENI!ybT&H4plLiHLujINI7bS`}LA@xGDtpo1IOUQM)KI zxA`OrDy#7J!5yuAf9+xNu5Qx`mVFEkyZo&MW0IuSHa-KlUy;`{5>1#};XX7H?ML(G zZ_ir0U#`QZ%oA@mClh=J7ZCbCO#B_ksh`VATLc$UtMQ?>RTZ@UvRl|-_D_i z%%R#YtlM6o*FmtbaTLu{4CC)~*g%m?^&;HS6a4e%q{cJo z%i>?+!;P>tkAQ_F|1061)9{@u- zq;z`CK8#^Op|>yOxGou-{i znq>7DMv+TLKYc+9{Uv|(PZ&OS?xgN`_*|4tLHhJTYC+xXx+5L`KsviBI+ROei=^sT zz(I=G;@P#ipbwUr;~YBiSN@aNpGn@=YGRwYx}vM<6paP<>ge;M90#_3`=obSW2?(F z1Dn*|nXR|s{ExyCYyk?}sn2;=o4#RK@M^(OM^3XuuAqpyzq#)6ub<1e;>`Ds0w%FY zY@u>xXelyM6huP;;oc8`vygkyvo%P#UFk4*QvRy`$cLCvzj@WxjLX`l2ti$byU2GF z9UmdgI(Ypno;_-!cP)g)fJ^<(^DC|D0jQ5}QE?YBZ8Q(QvQ`f|7p2arJQx{a8~<5N zPB!+Ym?Hw`X@&Xn9lTH_X$35~_z-RH-=#7F8L`Bwx+PKb0dBj3Yb{StR;R9J&?6%X z`l`kZ4F})U3&x=o)PJlhs`@)q=8XJ~9GJJ7f`vu2Ko~0HL2Sv7Y&%Kn`YVnBR{NzD z>&VSY8lA9tF}0=dZy01Z&Owh?{&6<9O7dxK6Lu-vpCHOILkpsN*4#A{k4=J!7>(lh z7Bco^FeaAWvkxNd&?>~MB;nPLGy+lW8GJ4zoSq)nIiApLo&q&6DEDS+PM4)&gjHs% zKs_%Hhyx7swla~0#9;prI*gRbE0r%mhO>-7b0D&h1Yb2SP121|1iyK7!Ey=Dkp#t8 zZgCe9x=+sWaFVFd=k?9WLfYw0Rj~4BXo~d7^~qaHs82T3qyE52$=ON_ zt?_8ER4;5@+?8w%*_zB~9h@EInX5%R7;qlMP4RmxQ+=jRk7n1{jFB$%37~ExCdx`F z^(A(Q*4F6#&Ku}e#3HvQp~t+a;lAA$p5vGGzzdO=yviFXK}f945?(uN;NZl?k9-jeh)&c?W_9|TDztPV87fsGWjN=Qer=AQQ88v$*xs<2JVhh%H0Gi^Y zGGXLZBzOsm$1JQP*(ul&j1Dz`MbVu^4M3C7ZCHSiII&ECm3eU=d{5h&?&Q?pafuj{K0)-A z7>a7Cd`Taj{vj>)9MZobQPf{T&M^YB3IbAAuOzUnXn z$D#DKu1Jh^|6Y+xGQF3Jwwplo_W>l26ii&z?5p=4v+6`H^KcsxDhR}JzKjB~<$?Fr zmnLV!GLHQoBO`}Fdgmcq^a2300xE0^+L{kl!GYD>Z+fr6H!ox9C8EC9$V10}*zb&r zbP9T+vEYtqBR*Tc?<6RDz@;(JuLHlH zdw9+g*Y<^8bpsVzF%ew4?)M93!Nt%PoxjekV@vk4o? z!_CZn`xJXQ*J+XP={!}{fbfXuoTL;iwo8w z|E~sej_&FHXxG`$NJPI|+1Jo+P*PS}Ajyi=UcwBhbMSW@t*Lwra~6B`3SsY{ARo*W z`-RwJV3WpFM3}tTEA4zrD9JnSMu&wz;(9{w!lfIr$_`yQjZ*t-Q7094b9F9Sba1y& zO_MP5$$?llkV?UL$jvIzsGDp7sFsDWoB_QX24y!fD0_I4Ww=R?LD^rM?)-AOb7|~_ zCDD;SLS85=r@tJ&X`*FBI(-(Pj^MjED|(eKk#(A5A_;iocv$Vx7fpO(~c_Pfg? zU5~-&>ifmGd}73=^l`7Sw|6-#lXoSNNyCiKo|Hj=lk;0f&$R&fd`o_+!pVF^PcK0! z;~>IRQCGcuk6$WTVL2GB^XHP)$}~pqIqN_``QZ<6F{T!!R8pUHN>L2rbnB}L*4+!pX6`L#qiPrRM_tUaQg~q~>3}j4~9+m-6VBMWe z9`$9UH_L%k?tj^p;(GDToH*k>RnY5~%ga~HH77=Ehz6q!jTtfZDk zb`egnHcoi8@r#Q|en%3BLWal3HzAAh3|WDHV(yrpee`ok)3en4)ub+4|75%9F4W;q zm8FYQKZv1l)9#d2^)UH`gG0T>=)<8YzXF~oVC0XdW55Tpqm=(p_;dcdzp}8*9jvm% ztU<|=ScOV1kRmBfNlxccOD;H0bbPb$_7 zU75g128XmWxkrQ#z(=64fjToyE1mcRZwQwE$_4z@NH&b<<-HOx^YK5?D;ro>L)g(6 z9>SxN#!*C*&h=;yu#T2qg$m6PacZ2g1NA{P{6;fVP zE&tt!eAX1nE&<0qZ#A-bju(1yRT1?)?Bu2uf3*7{q3g8_S);tyivXQq*)T}3-WhDs zZ<#^jtT?qhEA=8O=^5@$J_`Kwc;r?*Z19 zbW1^2qV-bAc=HZ7L(l=Cg6Iye!xUf|C1mx`j0&1Q+Gr0Z)Qtj zDuj+F_JcUR{qrN6nJwmL0{|X+0J0{Eps=MKi?^k+8tWe2Gi8 zGD{}!HAy9gX)koJh+e*Pm@}Yp*o^+;H5YBD>DtF#=>d zY48=+?!`(diei#C*E|?KJ|r`j@7D8!RPBZDVyY4I58qTWgtBy*oP)|(T|loX(qF|h z1~dTWCl$oL#x@_8a*l+4gpe~jNb$4im9X9=Yt%tU?+J0IZHR%_bS&r%Vn;cMaQe*Z zB*BY@@*1gaC0W~kpKZ;FtRVzpFmy47t(Y-H5}JqJ!+;0 z-&)xWnyGn73xb3mOp-?mrY=K=iAz0;41yUI(aXs)a9Y~9a|13;=Y_zB9Ez&`pv1Y8 z-9)D(qOS7Q$HXntPi$t>{|^kdk+OW>HUJZ^{>hS{N_4ocK_^^%a^uuuG%S)mywxvb zWB=%d&XilU5DTqNr6XgzuDJm_oKJX~si=W4+mlK{^eMEK_7%P0C=s6JROx&FeO^T^ zbR4uvoAL7nsD=C%T<~sS824Nx?w+i7NkpH{^N|RSn^P*^S@tY}gNQzRzf>563(Hjl z=L<6a^ye-IdeY1IljV}q@hjLc#pEWAlT#ro}9i7ot z&$lVf%3%=E8T($s_YyZRA(TZdT$didrrBYV9&)u)*Ny-~yE%v}wdH_n+?A-QV` zBVFEZZRB^P;SWw1Wi2=R7tM3Bk180o?fq{5UM?7gKSufPbq0Hk&p)ox2e8}!DYUjP!%B$VQ7r-Ae0EUZ7b|LAQ`}^M`<18H0$_IN>$Kt(BhEp@W zFHXxCN~9YY9W{>>a>eh*o_&2sAF#hKf>&<)+vXoQ0VgU3o>-zr;qxEBijSh8h_vMJ z{%xZE{_YP4> zO>gGYJo3x9HB!K)SJT1i2k~WvNnm)-2CC-V1Vmepa@t4u} z)wBNfS@27t79e{C_k$c8Hm{P??+z_jkr_?Bh3BPG`QiKbuE`g?X~04p1?Ij?L;tF! zh)H=Ev)1Tu&OxjV3s<}8qJ8$K!}24KevH!g9b%{!cFu1-iafFeq1IgD*SGv<{{Qk4 z$pAX{(AKv6EJNYLt6^wm`7Q+8QRfM#7dJ`!oz*w&lPi_4?BYiUnYda;R`{~H6|ll{ zxDq1a3Q$i@kUf+G`DO+h7{wVDOjwh$NIbRMICRnw` z1gi}MKYl>sG1O%8j&VFgLzuc-gQL?ML9ik!B62a1uoi+iYZ`Y6m>T%7|IK;jOaN8a z%}kT5W6V;DgdBO3AUB@@gLHl7&%g>dhL~UPGVfY+(Zeb$`O2O!$rZ!?Q-s3sF{+%- z(TszuLr+>}LVryf?(pV-1}4Fm(UUJN9v|xR5bIMxAsUU|^P||sJTE=WsKy{GiN24w zjIf{nC;tA%umlMRAg?q)cIA2*>$S8ICT0mOPt2X&*yE7ICZ%MY_18|7Q#q8P0fn(@ z@jyW+Y;55mW28D9vW!Rjamt}DFy8%yxYZWhN_N%3l{)17n2x6QZy3KM`Q-k-RfC&3 zGqHzK!J8Ta9p9G}sbq16WbuQm_#qa1F;>{utb6;{PWMm8{h0AN!i5K0t)E^WCk)wO zuW9z!@)aC;P%d|$(U=Mdh%b8XooCAo18;@2ZAm33vcz0a-21Tz{mz_NqYLX}L5);F zRl(PSUb2zEq*PA`$t7N4VN_&tX2qqB<*y{AwbMoT0aT zMn=Y>KqirtXw$_hqT!O+@5b%Uow9+-33RJtJYPlWEP0EtNb2GNX(zS6H6<$y_7yEI(#hf>8D=kutXmX@pv0Ri? zMgn&Gi|p0Qb`{e`F4=za|5pzAt+075l#x(Thx}a`(<=7t+dVHPR(K9%lW4R3>(h1p zH?YBYi4dfaU~9JF4>s&W9|wI`oV&zL{u|K^L5v%0wG!(7#1uEJygI~`_neLG{GYGR zJivNMjIOAYEh{CEUVIk*QS!+NxKzpVz?(iyZzCYS1z#~Lv+n&SOkG# zvnV`f7cVnpRzyn$q_xMK>wby_$}d5Vg-S_e$1pwzo8b|2zu+ShT3PJ);s7Sa!N@?d%yj?VZFvvhPG(kkAVlyb8(*C$4zF3T!6tX2VLzk~V}VvJYn`Tbm{15m z&!~Ss-x?Ai6VX3etOO$Rlm|u3=>gw5nnfyKIY4yc;r%h=C+s{`v~S`Dc8hD=%$|~# z>Ikrpf+Q8EoVIFNe6OCZl|B{7B(sBnwQnpTOU8l7=yYB)$g_zqqdEvR;Mo@Juq#l< znGA{F*+EZd+MKTnb_Q&GlhGWoc@Un=ytfiO)8_euu%D1JQ8|H?`Rj0;HFlw5kF{{| zh6jg%sWf3`;`%wFu;qQ{>pe`6 zRTM0=)Gf7EPSZBB7%ID!WxYNnS#* z%n`(!HrP5sKPIxp_Sdw=WmOF1=*K@#@rSQ6PdEzNtlCgpejfWJhUk44we3qmTyT!a z$??iP-JsblGE&ChDDF|a=LMb$a(3tKMKFeDJ*l`d6X~1T22n|Ri5t-r>Qny1_RLqR zsF>^xFTR~z>uvToy_kIAw_4C_LdtXwX4vq{&0U~Pz_~E)?bfO;9?qXAbKjRi#oGUP zU`zHH&k7rh)zYhN685`QR__66>i`1ci@gin)b}s83pjM z7HDFt&C>Nr3)9Q$U(2A__NEqoFR~nSX2E=#Tl$Pbu%Bvi4Kk4v5d18y|EtbW0GZAB z?YZPE4j1u(z}Hr87Vk`Z(&Yl?K!HGhw`hgu}@gTjZ#EPMdQvdRFA85RZo-dO-J zE~I%@peOV&kfee=#u9YOk}_?6`7aS{6j=}gbij`y%NqA8r5k^Z4OBn$4yJnF-c9+) zS7Ufoy?2ul2VHMmW00@-9TUf%zU(R1Q?gTUo;y`Ny~V|qg{lmFL5*W`J~{#-Taq9k)Ay!k|FWB3UAFHqa)lQe;$bj z$?KQ&oG<8rpNmh%wZsl*@)#LisIU8c%iHbmUQN%ZVWtN_+pCla*V*DfJ)Moge-1zu z7S7^)&-x7XvmV^xC&y^_VKTt#?$c!^DDhB{VZT{p;S2x&8jk;vW_;qXik70Eaq83O z{*!8;l^g&re(8`40?$j`@psu};f8jI!wWtiUB8&*RFy;=<4Ab$o)Rijlq?S6<(C0M zv=9c5-uKEV{;+~_!p7zT{F|=190dp)k@3Q>GJg|;lk5DzmXIj=%%Sk< z^V116&2$K9!S&MmoVM;tFR*dd0jTkH!J;kxNSuOcQ0fr&*J&v+#@qb+|D5`dK_0D0 z*(XTM_z>61p@tO66r?eU`eV!A_d*2s=@aa5q{nWfMd3URj?EqNDFPB3x=ffq68^{} z#w7uqRq+@8>#u>XGoux}HU>hSL*lO)Uxo>hU;& zK#av-k^RFyM+1Xee!lx>9`RYimrWII4LK@CN4?z${4dyu=u%nI_fxN97m)is{Qm?} z{`b9h9I-%FN>tGfJ@>e807O17S-An-KBebL8U|H5 z6SD?LL+3t`r&>DS(p~{6gVCd^;kH6jwfxV=#qa!dHeI;tB4JQSg}LshWl7PN?{Xw% z9@E71c&zaI7zD`3vzESig?Zq=VS-{sbyGqrU4Fooaw?*CUZPaSviNRI4UvnHLi^*M zQhmkLK5g7f1064OrooJI{#AOiQWRY8hrpvQme2p+aZ-;K#I9?PHnI8kpnAx^X)Wad z3f9#J;T1V|Nd~Z}-Sf8(j3hwTR}ot}z?)o(OQa`fWHPX~Ep+<z-DkEwxH(t&+?0B=?kF|}x48|3|eD}C@BwN12KpCY1z2+&`Wq~Iukic`EKoCD5mnJgl{ zmevnm=M+wK`Q&8~|2cIKZ30Qi^kQK7m>!t)<&e2+1f=GEI@n-Bkr^iB$*Z@wB|vPX7{=pFb&s~c#X{>t zR;TTgymvyxPyy_}J5@cr2K1=!Ve1SBxl2!g$U%*rk*2h__~e7f%$#Y*7u)Yg|^o3zj%Q-=)9>U*|gD1Nxf$OITfI0HXT(l!klX5v{JG*O0kQ1@KK5J7X}A zxn-t!J3Jd72qJ>vTQ}!-p0!&eGGEhu`SX1MI2cQ&2_3^1jYZ;n19`vri2!QrgThwm zOLut35JWC3te8Po?a^~~MmAsKnG^s|0Pxl}+h)mL3D6Y;b14iHy=ocqG0)TG}HQ&a<#psWn~7lw7>uQ>PtCw0fGa#?1*IQ4jm~Tz>ARrWF#dQD#bc2I&0lWd)rZF$;1>)HM;N_^mOpO`ZVFxJhA1oOi29iibf$O@*4$n|5kEd#&n(Y)6DitTkQ-6 zMeS+G<2n8=7ai-_Z$ACMt#zhc6rQ-6)Ct9=5P_*BGCl$zOCL*lDjyFmfim)-2cQEc zqqJFJV$dM30xFU2ZsPVrrqfg?=E%!0Pn-U}P$RG|{ne1Cf7t&-)gd z55y)P!#l%!S9vamfjPa0>gV>{{`G6)^FiFD5Hjfl&?#?WF6FHXiINp)_2b4nnPz_3 zL;GOHiH@!=()9BEULT%67>rTyb>YhzGb`)*qWAc~!MJo1qkJ@U#o5hG(d%D!rB~(7 zw+0)}mEe;#CF%2&seqSdgH76;HevsCZ9jFAgAD-{%N>kv8w?UQ6MFpqR_8juc_4jt+^7EhnpF64m|hKQ(BrunLsgS5Drr;kB~JsQ z5OHCc0T-XWFKMFit8Ml#*2sp#PQiHfI62MgJ3fms2fOV&oo#hEPZAW9zPVARps8OB z;O!)~X6;QSG=CG7##M+hw&K|jUiU26y#umNXxECq$`?O}clvg+WG3KQ|B~vo?KDs> zPEHf2V~L>rZdE)yG5StAs%iC4NBOxFMG~e+?*Ayy_4Lqc^De4Pi;#CiDTFklVQU{= zY-E_Y_5S-Yn%BZsXaxp&fR1$m&>xwcx_%vV049WBgPtpKoBJ%mkK^~+d^M>nbkaS5 zBP6*~f(L{h|Fm|MTs?KtAJ9o^{yv?^0v~n${m**a=ODzteoL{obn*Nde|QMX-sq!? zec8uMV-KU=Uik;AQq|I0cc6UjxE1kuE{u{sXtdxx)_{0e>x`e?HpnGH#kBH{^`+C8 ze7iDyl-K5$_cU|=EiJB$T;CsWfhZnQS3A?DLDqQo2DLkb){k4!GFOV7I)uiJx*ZIL zFGPOM2Ql*7?ENW!NyylgG^4mWarAo1S%~6(A5LrchwT}OwP_A>>hLbS0|byNptq-_ zs+)Lk&I|WX55DUsL%)x0x~z1S64KK-KaJata@=ZjN!ExjK9K+Au#ZMbh)o{T&##1j z4Ko}~hbiY&>9xIO#tF@r7RYUh`%1MA5z`>+zp@kZ8cI|^NkI!p z?Wt$+x^*HUSDpr|#vs(O8PQl`E^#k9c<3zHudkD6^b(D@V*KyT7OoQXYGj8|L22sT zhbD$Z7;L5=PoNY=ytbAb(>(vqz4qRnOj{?ftLzKEQ(V=LraTkt6kcGx)%2>GqDA-J zEqJvxq~^>r4bf=8L2hYFn+SJ0{5OU_#uZO$z)XtsSv793JoBU4^Cnvprt9E>0Txnw zYseP0lKcHgVQjvv8j5|pcSRM{jg#p4;Aftjy$>nU?TE#`Wy#bNpZ@LUTs#lq5WF4t zJeXr!8h9LYIA6YKL}sW2jWpHE4F&gIkc*bF0IE4{guN1R zCyR^)b9bONgYgaHtJ@XkoPDUUD1^mP{Sy@T6J?7R&xZz)E{QGx=uOtWZ&_~OKlG! zr}8G$dxLAj6KUU*)YS)GK7Ym|qk_)rX#ZGBkwQbuQR0s*E;IXZG{*c6Gw z5tTaMOxz=52wzfDFUM5e??&tYp=+>EX8>xP^!H(+*oEYUJpyjHuC_rH1DsudKp@*YOp*D^6Q zTZyk~XZ`-9%M+vAvsl2jTuPhkWOd+DTeU! z*!hgS`!$lOf^1Oh1>2$OhWxer{l7ij`CsRG%V`JDHcNu{KT8dyVFqHinMC8gpfx=CivC1FyS3MA)AQZghK&F!XNmioe?1 z-nCm3dgqK7>Mml}RJKZIMo25`2mJAP1-D{o9>pDi`H~YA1s*>~Wev58+{Mmx@PRmt zMadIy%TesOK6yoHQYE(ks_xG-7;(Cy<4C{0ECX>uKTp*!8CR}_rSeMSha$vNUcPc) z-QN&k^9l?FKV>CJrtrL|KwWg|v~3<xVZjRXAZJureyDeK@PR~^Y0)fCHrV15a7k`g%fB5jY0Sgu<5 z{2~~W;Sc&_FYdJ6wbyZ$gb4vHu5}d55iGM!C5Fk`&{j(x^qN{EM(RhhQ<=ttQh}mj zB`XaQCPk7TRx+AYafogg_{kIOUFdzI{oCDD4Gc(mbA3ocvttNhH1k4abdKhO!7ek; zL6SrLhi3h0lTrKht-=vJN-wA5je=E#D`Z+86tm5cgBx^LAnm1@#l)X4bC2`u-ryV6 zYbMpzjSYmMb|^KvD+N|_d?g=3B=cyVTKD4~yyfE`ZLZQhqWT=rD%U2F8H?EcS#o~+ zFTJl>Xipjie+L8EP4*M6bPc}`(NyWJ*-=*{Co$J#b-N1uawEnSH&+{}o?jila&%rl zE25RY=wlVVvU3p#c{u31@P5Z>aK%pu=1Qx;r(|HlDc<1GRkJ)SG_dcli0l+?VQH{A zH2L9H%$c~TeJggBAuSe`q@lXw+cF;5cp2|N7Yk)%)QC?!)q3pAyRZH`iHkpCo~ECEV zp!=DF)HuxAV6ANsfp+9T>mJg|eX$^MQ_xjPy0dk@J%Mew-``hCIoA zWUYE3yN+smm7@@It-17_T~YauoGpo#FW5*~T!Y2Evjg(>o7J&Wm^O@YY0$`dofh_W zg$?4g(4Ks-$~H{pEsPYBZ+YqN-$ED;6+|Acr57N<`5?ZOEBua6>LH6lt7=IgIg27v`H8-W$dutF-t9gyyd$b6)&R zjD`5A&g)W@?+D$AACGUqQ)9MG2x_F+QN&B+{F=e}1{bc*nfhDcR>Yj^l-niX}> zj#sN!f)hj4oh8mMCzh%2k|Lfz;1V5j!lN%4!Y{C%hxg2DXx&e}KpM7v-@Pw4vqetC z)<3kp*BfXK6XOsu!p!x?e2ux$GMfm&e4IYz+3xxDD(r1l+nRFHZluzmK6Cub5gf2 zX#Q|3F@6sf3luFWxWE67?B%nk+|_d}_tf6`{Vsd0wnmC;vftC%HMQGDr4&Qau=bV) z|F7l&ihP|s$cO`9oP|wsxRjS_rl`P2LzjPDI}P!mBkqA-dqJpqvYWeBR9GdU9D}o> zOE>Z>kC-6UeNi>9$6cNPn0%F9qN5sVnV8HuTnFe|=d+{&+&JB6?Be(kQa)0ZgC{~} znh5b&1oK<;j0kgfY-D#)y)1&q+7=R=&-0bTIzpE+g~M!=eq5z@Tyb8vV&d{ws8LKB zH!HKElJ!krP)C-Gy`5zdne<=HCn&ke-d%USnG~OQ>1*0;{idcL1<=FaZw;<0(3!7hDw85TV>!;MX?r7`(Lz}X{`g~Za`;JZ2opfxw&ZD-t$e63B++xn7Z_7PHXyGxj!V@aW=(hhq@7@nXj6Qd|Z zz@C?^AAfqWYj^2e&fR>^47&49cxCyuE|bou7@1hai(20Luf?Z_wzzIf`p)?`{}XFT z(p&4}u6Axybn<^CyQ6Bb=QOT&>Z9Z9rITdzf8wVt;fJw^+T&CnrrO7E$bU14#v&4% z3DRqS%*;_9i{TBKsEgh_e?5B8KskAgJ06@BOR#StntLtgNwddOTxRV{`1YK-Bzm<{M!oVtS zC?usq8}L6u!A17k*yh!f9(1+4`kX;ZpfBbJoRcFkOn6<7#DIk9Ig=%5q^Q zK(6uA%e=O3@gId$CwuL4HO;6|lW&9RM=95)Sf%ET6-aA+Qg*U4V-J{hlf*(UiPtEHmwxw~sB&sZ{fzpyPVxfW0; zFpx6selx`dG+puG4adoQ2XOIyR1Tz>T(mCS-|(+~-o;+1{UWHsq_lg{f>lSqt&^=( zs$jKK+CoLpcki9hBjvcsZv)W4Dxb}d%TcjO%i3X^yKFZ%PebEI*`Cl2K-(Sod|xPL zRU0t=wsOxygr3&Qwpp@ylQ22B{gV|ebq=sH9-qpxP9QYAI61qd!QV+F@@yUQ2@tV) z^m5^x==*-VUQ6GqEs1QSms^I}ga=K2kt@%cd$ULH-?7a~UDZ|v)`YRby2{-3I2gjRD<^4bf0&%y~i2tP+$Q>lADQWU-_ieB6BXbw~mX; zn*02V=jN^1C1DuxT}cau_659<1lS>YI9G=Iho5Ea{iWhJ_dG(+6b{5<$7+KL>1#j` zs&K$@CV_uAU(roCGsjpwEFRnAZ4WxLsh;2OZE$Vwm0#4}WA=o|&ch9-?P_1TsJWGAbLNIG@KO_un*EDh3`;VL)7v8k zpmUpTjKtcQHt~VsHhF0x0D}`zS51ynk?nVD)vKopP}&|eYX4ok1ho`6vmCm`?h_Rw z(R#N1>s}y4a0R_8Vs3A52kPyvhk3vc(#!8`(qP%K_RX{tF^ef!w(+tAlnT@i?&2Z( zR(u(AYZN$9_#9($p5ffoc!21?=G+~J&_FML7T@;Yu*5^YO_zS6nz`jusi~OB#k9N` z9Imkv>KG^%A~=y@9<8gb{W%z1FI~^@*YTNUT**8?YCg2+!@nGVk$Zm)XQInZw$@@p zMU@Y|x8eG>W5oQpb2u{=H@zxC7ZRh`cc*j>p1||AQZYhMAfj6-bJ3Rh_ryFZ&0JTM z&PfZWI0tlWFd(6jzW7V(!c02FKYc|fXelv6@=_i}q98qgv@Ojel*yx~S7rCN; z^SmF={#agv+73Qbe`Ng|<+xA2()63lbEzJ6dF1tDy*M?Xjhh%s z627nHKFDEq$C}8suiiL5@(k!U3sG9*jjVe(zsmQXJ%rI_=4Ad zE}@>7NP;{cR+cHthb3mz#-TKG6;CO-zE)`Y*g`h#>x$1nx{{-es0g{Y(YxPfWfYNz zTl$51qpX&&=DE~g_TZxFVGAO6HR=SpSTQZdt*h9yCb=A`%7WeMe4>!EeX8#R&ila$ zvZwFk#r3`zh!FlM5Z?UXbTY|)P|7%x}qm#bHh9Ib7{OstrFZ!WoFCHM>@YB4r2-|`xR zgn{Nsw^gv9C@B_3A7q%N6%F8)pLolu?tp|X=q*`{MAT|+{3VYwg)=Bu5w_rv zr=MpIxb5E|MV?I(5lPRMf77F1$pSE%=^G7$D?XEFVtAV%|vEH z&^csgk2AH;i%rS~SQp1NTK;5mAu(@`|?hs*77mLMe;j`_Awe ziFGAH(ju`Mnmw8oiL!TNjO;AcN_qJ9Kr z&s>6Yw0p$!;}!pmjS$+ME$qb@R?4ky=I$QspU3sLy8AQ#-G3=iwZFrP?I@g5lXdyclK_A1Gi4mh=K6=}@pkup?^CDl+mtEh>Iig7G}&>}bDYLvQJ*mQv16Nl zpUtl!S(}Nr@oFqz<8_{=Hjv%?B(r$FEm5rPo=e(lsE0j?P~a5pS+NKei$dF{|D5Js z1qb$j;H#I>$gyEu`aQ2U1FSxZ><<3gSUGy|>&N%2i_k>zxL!LBr!m~iBp?`j4x{)P zsU{wtH2`jxQ%!e8*o6r`&Lhj8&(+Yk(v<|GK{ZGpM+tqAy!4h|Rd`mDhgWQ$FC!i~YH8J#J1-&`xE3AM^Dc~}MWB`310QMHQF)j>blaDXki1(j} zNYBo@7~Z!%?h9@Vy`P}#&5}fH!N5-b>R?%j&4>o8;6N&w^<@j>uJJKpseSU z!xjU(VHKXRTBJ?`(?rDEpN%*s_%4K6tY7FXYImb0Dt5Vay$pT+0Zz1`PdTwR&VZRpZpCh0N`&o4Ewt6NIrXS?7NJ$)<X^h!$6Jv(Z4Cna+;n_Dt30 zs;RhX(CErOms7=GP4(S*GyQanMYZR-eePDSvZt-C9t6+?&pe1@GJJroiAh8hmR^qA zfJE(=yahu%5@YBq$TxM2q=_Oky^B!Y&UWU2+F7@412j0F1Vaja(rR%D&l1^Yd$6N) zzqjGQ{=~!Tvvcp+9;ev^!qN_t!Lfqh(+@p#oc_XkKaB+0Ht%nZI7VM)*Anp0mT&jk zbH$3^d^H1;L^JyOMwkX`lz-u2Med+Fjuh|upYm7;Tmh!_$J>dv7z+P9R;oDG!^)EE z{ED*nY>lr_tI&GKgeI#<;kxB;%*3nsZSd`s6_wBES08s5+|i->43WL-qz+cl!m24( z?xQn3L00phD4P!kfoc*l5lDY3Pl37RFaVI@O^_nGhUpBZtk-)Is;b%z)88*ldR~c1 zNV??HIp86bbTKKpbWm~sR>4))&+MGNB1F7w+=ac2aqNQ!xuUi zSHjkaZ_ZL@f`41Ld8vMTL{`*>l{=SW3P;f@U8Z7>nD}D{e2+ z!(csf`f0X8PidGLKcSHA^5I><4&TY6{D#e1#@wIFmOo$Y74S3?mBKT@g^75R;ya@$ zt91n=8vZLua;TQ-E%Y|>VAIE$q#uJHKMC7>U`SyuWXm~Mo*;*+kHnmR9sHa3tc9AJ zX+3$z)Z}Z3%D{3^@m-0iKLS3FttnsT)n7Lz?#sLn#bCXoq5o;o2h;zr7X9H5HTq75 z5M7byhjV<6-m6cl1RTu7X^Sz$@3fFQ3yI3@eI&@X=gQ*QJ<%vyP){#M$}_F#+bdrs zWZ3!1UU0g>%u5`bGKNE6XS^ zCsT3=6e2p?)C8KNw=#4ZJEb7gA1#l2kxCoR$51Cj#wa-=;s_i6Uc9{w&)*b&^2t+4 z3$wk1NZUOWF8bF^=CbQh%y(2*!Ta5HsLEP2*ok!*$mWb`V0Fp}2%@n=qd$I#!wa&Q z@`^o*AUzsBlM1f$ zJ#IB?mY{ZM!&P!LNvwD?K%#FRp}|#z$SU2T%d*3FZ9@q3XBUN9P25{{b~@f+io3@2 zL4;RmGMSaw>%7acUV&Q2R{O3m$`#F1JUtU38Q00}^X=d5<@f1Xb_(s} zKF~Nd8K$&&m^%vt+Y81=g?l|wJp5caKbs=3XyR&MROYMvdf41(uhwqEw%dkB-B|+Z43#Fyt_5;D1HFZ`0hu_Y~y`uAfap)DPhz zpRA|Yu^d2(r<@!_CW!RTy~A-urc13igrrOkSM-n=0>zfi(YmSA^^hL#x6+?|w5trG zUr507jDTWJ*XafTzG7}NzWuwUw3IMj?fdBYK2z}Rr0K^>glB+xHn`i(K7bbEc35+e zP(=6?n?8MGT2TmZj`2++G{DKL2;n1lIu^TWj|Kk~M6{*8bG=92L}$+m(MPd?u5@Xn zoq{rdLQCx|S=cVL(?o%P*KP!Ij3py}k#|$y-t1TkUvHf8gvK67K|nvL05&Qpc}sk- zDNBlg(f2wc@qR&8|ANW@n)Pf#t<;zTw~@*Ml@hTQDSjF|J$^27hsv0RUlLl|%YVQ5 zT}h*Igc?CGK0)#9$DH$04$S)eaY;=;&dO~-SMKJ~UEhGr#o<5223j5$9#p<>KHEvQ zfB1gm7}EX?DO@g>?hg}ua(9zVSE}~IDJ+v;7CJwLWLNrnz(2;P_hfX)<2`UWrdwNE z^DUpts?&k3Odw1k1i?m4N-Fgtz8rC|wsusv5JL6?ma{kEi-`y8QIZoW1C5@2-;D5iDPNAZHjBufdV@+B^8sL|2m3=3sN)-_pJ4%i(boi!7nFOPOus5+kQcWF))=sRXv{9qQk~%Yo z-A0vf(jH6AlAm!|UifPl8>MnsaIvh$mz0{S4%D?dKEJ?Z@8)VY+=Utf@^&rV1Xa33 z=ADPQo0@%7{Zg8NR9rPJD*5+=QJ5?U^!ovmiVhV#{%XlTIIyTkhhxNw79xOJkrL{J znnR-3Sfha&y$5u0CjF?PAtm0n|6W9BU`h*d=AcM9tG`e&?NE5lWq{b)#^&c>Jmpyz zEc$$>av=PcX9k~3?x_$K37xp(*q?^Hmi6un%WQ@IoI+Id)7^o~2=|@QnpdQ0tstPF4O~w5KOPC z5gGX}s&;oj)Fx)tgY^htq(9@)5vZmX+Wfjt5;vf(&5D)p4Z^nHJI=>j>xjT)@!L)* zPkAs5iIWF@D11?23Ugv^Qnp@M*=>DRvEj;OZ-?wr5FEYF(=PC?$IvPO?&(~l8kx)7 zj5Aj(kn3$Yv5{dsV;ZRxzq}9T0M(gg>#F~KM}^Ff;Lfj&qkNBBZCoFCr*W~4Q?`ef z>ZHs7D9V=|Rf2Z;qflxYN#?Iz%l#6l(XCy`{oNq6O(!{%)9S$rhiV1j`3(lNDN1EJU+_=`E62 zrKQS0E!kDyH%%G5k-*zG2iTy_EJ}Tk^#6bHAsl^Ae-(LGf^itNkc_l7HIb#()YR

s-~9hyj`c6r(%!sB8hz+|Rn@zyzOh&)yrs zGl>L)@-JY@&`LpvSkJtUNxD}Ig-p?He^ED z{Wy8vb`ov2;lfwAuF&hVJ8)1tWZcTAB7vhtX`&aitJ;P1r?nZRRq7F?%D%=+u1;ED z!ZFAv@9Jdh5yZ*}8}&hq!Q_%ifIpp~-}2}V6djeY3+OhrHoMyiGtm*< zmI{S~@$7a}{6c;J`&yd(wY<@MAk_WA!8VmGNoRL?&O;Fa)_EVPl%#z(Dkoq067haY z{J0m=-7T&{)i+(DUSl!*&5%gef4=og)oD92HVqjE$7I3(8MDCj!m%6!m9uoyd8xqF=FZU`tMbNEXU7TKxE3e@v2Ld{QzYFR=LJg+lJ4piB@C8%Y85fs=x% zDFg3?z7`u~TS2?Q^cg!G(>(IodiWI?H^9Paj7#2oS%?4A$$@5F2ISId5R@2iMXt8l z!dFjPw3!1;eqy`-UDjCU`|YMXtv!MH4(|`SL=k4hP>fGew*v#)4`*gePJ-(bse2=q zhA)_YCizJI{r%Fx-Rk@RSAu*En9$A(bo;;13b+~dmc7xutB?>>JQSN$x{#?_i{apo z7@O2xub)Jk6xU3bGOho2=2L6%P}cWfWo0S%^&p)9w6wv~%T`F9@p`C$|4Ks%j3Xu@ z16aCAPn5Elrp?h2p}tKzAJy{NZ(psYbQ$%cakF|Mq_!>2x3oL~BS}9PjgLxWY1vH= zG(E0T-^bXT38y4muy!!dj(Pa%39yiYY2VAJl2CyrdY%)Zr>C#suTn8JB=!T&=&$Hd zlpid&4xG=Om&WefKTAKDDrBgDA8329L#&bIhh?NU?aO!u9T`_R=_%?k6_eoIyH6zv>Ld9-)x$3aj=>N`w(2|!6hJi{R95N38ZU!j;<2zk{wH#V~1QT=v!r^Z&Bp-{Xt?%J^+I?5<2y)!wu-KvT;q=WG z!4*233y3mHH5L8-X`z>3%R)$G-5LuX%Z88Klw`walGRyALy^FT*&~4PVFUkB^m8a3 zL)8A85kg~Z%%NV{nmrGJHsOIQUz$7(@v_*4%~GiQi$;)2?PVvP<`r`>#4tY$!kyk+ zNqc+O!b&7G=m>N6WU?Fp`F!yn+py#P&2_X=>{@e_b@_2jN8X*f2D*gpD$tEQslczX z)&XVm&Cyiy<6@yt-^=VZ7mg}PYLX0;P*=Jiy@k}s` zH2c%u2w%alibVw_gWL15l&(08*B2UQ+;YjNZ1yi762gA#%uPi7%V~jjvGN%s4kb}m zSJ5rW!Ybr>Wew`eAAmt=O)(JpWnifFw^o#I8Ii*ZH-m6iR#47RC8dSo;u^)Ylu9=% zy8za^s-e0fkBlu3KJxi@Acr)CKXi|=_lIw^{40b*`XqJ9Bi#!G*5ie z@x@zKurneIhvj|fotfJc)B_5tF)^o*C_*-vEMyH4y6*pl1ptQ7ImJl;BIy7Onsp}m zz;9_8+iixr*{YCutWkj2>66$0kgWpPMyYBUalXHn>=fg)aRca#AfmUJT1z9579bvw z5)g`8_s0-N0{e5|gqVRAPQF}1wZjsRNi>yUe&JRATn=dFWnskW;@Rc^JbAY!%t3Y- zI2bjh)0?ovK0^G1XA48Sa&&mFKAxqC`TqChO%1^tpl;J1!2w{nR@2kM&p$a#kp6@H z_(Y$cxEfG>8yW2SQ@+}exx4q6eAOW8{WtSr{*yz;5i`1|UYZz#fGnzxdPS(>`6o6q zF*~aRrKlS@IS|Q-k|K4JW$!u_Gu=ueXJ@b6Q0$zgm8tyb_sensay_n?f6ILQQh^$gpW~|j@x2~bnk|)T1yuu3EF)%-F;#i3Mt zd$`hMOM{MWrQ>p*;pSTfE{|4ah);i!V+H$v$Wqr-->HTe8k6$7FIvjw0iVMN|=@cpjmHc3*lb<&A zQpD?+B7D#?PEk|Dp|KW&4#m^t34wl+`ZB%^%EkVFCJTCd?>@-j#;qlemhkDK%AeF1 zOs;dZXJGkioR~>xzCl)vFFs~7GxEXfsM`D30;YB#Mg(BaGkpu)O~eNlvP%ee_HqZF zT=-1l;0S_>x)!Xa7}?&0U*K-GzR@7ljp;x z>`i>1gNwZ3P66zVy{5xHr3^?9n+1jRjrXVDUgVG#ZR7y)tO_Pf!p~Rf0)eG6tO7lZ zXLtiDkQktgr^AH%{@(9s1<_r9W<}(*s*&2`1arXUiVXGZtG6!`i-cF17KuXg0o_Zr zD35h|i^;DXa=^c6Furn`gvE=2kOL0+-nq0TGoc?{K8<~*de^UHuK%?v|HOwx#O;q4 zfEfJbn3`KPdYj>j1?)sxIr?qCSTDEe8*UODMPm5Msfu;LXZVe%t_oFEObss>DWI!q zXA9oe00P>{-ImRdjnU^4`Wy7tRG*!nWHmzV;kI6^*>v(1zgw(NuIMr_UjCs|Yd$pA&tNMd2&@-$-~Mj(<~{?7tXDo9 zTZvE1v;D`9xAgS%lViE_4RoF(dCzb>T3j2`$+Kc2uIA651HTlCrohpd+n!D2h|ikZ zQPoJPQN`M~qlg$Lvf_wV)p$BD2mX>>?U1>yJz5B;a!d_GhIe?r{~ZJZ$0fh&LGlugV<(5yo*S=T?Q$(<^Nz+PUErSUcdC!xys;FL>eM*g z*@_<;^Y8vRm<5wot1quPqab|zPWZ#1o%Ti&MgmPXbIbm}Y?Q){70F<-hDf2p*8Rca zFO1{RztU~7IV~HDsDDjVnsDqgI*L&gM|X)8ir@cPjcMp0;hYlGaXq5?@r9e>@smKt zJg|3W|KNb#dYlZ6hz~=4YwQ{lD{JMRkPSEe)UdO(Ip%V?jKpvz|M4E_e zv6oh2Bp3J+XCldGmFX~$(N&w#(0upkpn~%{y0s?kQx|a$BuUfl_tNt6){(cctdz|C zM>?D?_C3AC#^xZsBr+6l+awes`_I=~jRAH+uPG5|%h;)ODRFR)hW3$FWj`6!G3Gk; z2g6DCN4v{kuSZ7?tk+}48?Dlr;kq$Lz&RRL!$-;=jhk;tt5z*&0`k%T9knHmA z8S?RHDAp1K-}M6W6i55D4Lo@>Zk)bQxQbF}x&+cQ-&^V6kNpc;%cxvSD}xV@w1b&! z?8nUF$m8r$?oP4r*OlzVHMhx@mg^VdKI`d>9pyc#wuq9>)qj@-iSGRKFH&zZ_(F@U zPx}wwgYjnql8||t&=ub3xF{Fhznwv8<9~-bV7cdPEkj)ry`soc;v_QLfHU~8(hFV| zppm=t2wR|@sXHY-S03v=TW|;&U0b#g%NPmLkC6*}o=Ss$j$8^0Vm|I7nG;9?ouq#Y z8PXMijQ9HS2?;+qeB#VpnWdzK!H)1?!~i+xr_ z(K_mXUF-*CqM{{guax^B;c{AwbK|bh?WofXTbfH=dqnw88U`g;xl6qTuPR*xs}sLR zl*9X%{dbTLwlV$$uZW={42nuNBMkpc!tMc@lmyZSDP<3&W0a?=NR&1}Rhd=BhoY$! zX)9=gJ?h3e3%>KdvSxVOz%c6~+z(U0+fU4wj1I%;Lir&Zi*$784csee+g@Z@KJPuP z)j_qhk{em@?sBt^kMomN!{jh@PL~LMx+b|{fK)1m7|c2)q=sat`^Mr#+^sH z@ikDcq2x?|boG-33)Xb`$<*5U&wKUkcQB!t-aK@oFES%bZF1aa2$ogmG5wnYJq@_5 zu?WTVf%~%s-eq!Lo5X5suY3FJ&>I>oOE#a1>2zY-_qf&skHZ@)*BvxafB@`cHAH10 zn0&H)Hsa;ps7hPZ!Yc1%GeTkOB^Qq%-6x`x*FZGHu3EdSK&C+QFp&}ONR*ROs4WP%=Bq0XKYi93oHi~-_NE7neVl`DC>14m;sb!nypxr%UWnyC+4E>sYw}sE7}|f~RnOcsM(Mhb(%j8UDuU zO$+nH4aSA@h~Es?;3kEMj1BXa0}MB-?h)HoS^CAs~X!AhCMg?Xu{&2G$A}aoayr8 z`|Yk?x}npw8F}d7Z@{YDNbBYoF>jyj*gJ$wXoNqaXA+#7ZM`xx%C4LV2n-XEnY}?) zp=((UWx2xP6G!Py_8;4kSSgGyc0WWjr*537fr`PP466H^1PktjwCBEpH`Nw(yreQE z4?Y!PMx!*nK6x!O8w-l{?m(?_bo6H}dW=5sJye=rYbk;+fA{8QMx6tl(elwDi8z@= zPm^ra%ll=b(c@x#@CcUWt_?SG+Z-{K^i1i8hIvbC9_I}u z;w8uZz)0r!HX&GqsFayJLfP-*`0Q43YPw-teTCFw%aGsiJ11^8?uLr|@e(iwBYQ8o zma2xXkUR8->w{1(^KS9KKrW=D19@`co^X8 z?d-{;^xNTHGH-V$<(lCQDy0D4Sj>2R--SaP3A!z>llW!UsAOS{gA^J%iezh7SQ>-N zz@&l78zQVD36tl4_qh%68hZPq$K|K{EI(>sWRXg?5un)@j3Q1-fFF1j|AyL>9eAzM zXp}d25g-$+=9}oO>cXF>=6{0s-keR@JM}q1Mq zv!|JX(0e8OED$Enpyw$k9U@r5j7{>QqwOv5o^@f8XHcAOeWxF7l+2xex2pPs+OYsv z$H-$S2^^6uA4Czy#;t2?+X%P#`m<2D&KvRvYrt#V>D02L)T1U&*BejPF~$p6GF`77 zZ@#!ajDP?gkMR|DCxsWM9OXruzs|e4h>8|YfVzX*)cNCTF-A+Uz7o^60MyFr#QMs> z>ROhOOtKwP7D`yK7HPHz!|15H;H;x+HV*)QBTe=jkB+!ar{t#u#}{Yt00ayB5QjI?)GWZo`5!q$jq8WL}FB-P?%IN={F zUx7w>ck`Qg?_pc*Shw|zuhNd3_>9=UTvW0NVQOa9 zxAG=WFIZ(JAs{dW1UNfYHn~FzKy17qgNm;hK_?3F2kuDZ59mlekmlR6?trVXuG^|- z2j~ssJdk$n7&O{l+`f8b+TA@y0RE&yGOjEm?So9& za9I4wH_Ex}?rXe1h@jZYMJ#>gjFx=8yBSzOaqe?8-!2#KSt~1!`BqsM|NP?$k8Akp z!x+n-1>iIR2$9$erfEOT`o#-u&;|0c5B{Aj-+y3B{AiZB_;Kp{;JQ|jSm_xT%0<~+>&I&!?+FYqIA}Z3L^uecu98v3Qv?#%<>;6L#CVqEV-_2q+(gb{*arFs4e~@#| zFOb9LZ{(Fj9Rm{y(X+{tn_@CLCp0!;DqEy}CH|QN z&~Q|ds-((m(oQB0V0-l5IFTOrhF|;W_RsI6b8wAXSXIh{)6gEy+JaQ9CyDwQ<7?xO zZu%KvQupP}n@b3LJ81({8v$nOusJ~@Q5cCs|)%l)Cs()K71`|XN1rQI+_T3*rV^F;JA zQUNz$m1#{Aw}3ZQ1_HvuFnqQsk~x5E=+e2|z5_Z!7_bT~vJ}9SzKOjsrwj#KN_fX^ z1(`@Pjj;^Spa!(!It@TqEZsk%SWJ+*<$A%y;}NA5B<#zkWEhs0)^8fyc_B@l$-5l8 z=nB{r1os{F)N<%I?L-_%vLBtj)w6eWCOH#ysnjayIt53Aqagm9qQe_LbZn*0YK9<` zgz0&eo!(U2)@ekN4Lp^Z#EoT>68GcfiH**;C!ztqSah6ADh~s!aeXtGk6f^A%<$c` zWa5`C75dfB@l&cPi@*Q^B0L?LAE4-evna3PfCD)Nj0Oac@0Ml;Bat%kAeIuQf{f3w z#BUIPvr%+BS2|x{0wB9Xh55;k@o`_Va^3-JP_jjf;SUuR$9c8#4mTEZ98-@}McqKW zZJlq8i0Xf)Ww5q()JOn7bpoH*ki_ScO;Ytiu<|P#Psv^9O-QV7H;=fR?=qrg@p&Tp%Dtoq>gdT(VN~lEJ6hqK zJ_y5f;;(efBr^#ND5;R4fugnW{*vAj4M~Ja0Ri`s=HVJ!fHmvMBEK?v59WABnYQYhx*>8rorz!;#pwpQd}#`05h zbTTBs6j*>t7eA(dFzz}?cJVu+OA~NwLI=hAcA!45s=yjOWeeALD*)H#2NE3v`ubfs zzxzeh#>YLZdmu`HqX%57EL|p$bEt?~f(k7orzzW&2zS35;;oUQZ2Us{E|h<$WRHsG z_m@y7YELD#`me?GC}qO}lDPQ_h@7mNgo z1_k~r%aFuxo`;V;OuARG89*!&!_>tVs0=(daF2T<0L9h$E*C?|0LP=vWAgXKj@$Eb zAxLOke9y=i6+~?=*GlRd&=S%YX3L`BvDt=)?2=^)zEhytA%$cy*2)Gn*nyuj)ESuvdhlmCPWI(yJsxUXK94hjkCzgxRJU8_jO%o zGDZHSl~}aqcI?lk;S>C?u2Ml@pf(~52mTjrGQRv53mX2IVytGbAvdOZJ@hqyXz@Pp zJB0vuES1EQfsg@nsFPJ}Op(c_a`4S}|CKGViA|&K1Wo^7a3uzmQar#KtD2WvOjA2x zGR%@vtOxz4B4Jc7d8YBdJ9MA81WrC_T!a(xPTBvWa_9P8_S8VJ``_#2duB%!8t z3uv(eunl_A;I~d0Q_IU}$;0qT@*&gH)3W14mn0LWU#0^tdXNX6KI7~sO5aS@-EADV zQ3D!K?%%Bu##99_^5`aAoqQ??=zKr$y37Un4+;KB^+ML)L^8J7oddp`(z*Lnnd*^h zPsR{5djr$549wT!-_oImLebIrX>mgN+t0WB{QNJd)ZK}^8lXI1B9oMqR9se;;7h`3 z83on|zPniuLO%rvWXL%=v4KHUVcYWo;x9zLcYCyG_-wBLsH%)m=O(ga6CpvgwC7%F zS!Fk&Q`C8jIGC`LI>q_r+Y@O!4Nz^x@R0yM6|%60_I-|Mfab6V@ij2Md5Wy{CMqb zfe!gfQkMpX5W^@s@FVO7Xf|cwa|iA3zrOT3X~!EAzAxtr-QTx`d=mG7R09MKZ|9qw zNNelrRMpi#O>f_0NkedO02aNLfvsetn5H$~`{HZV?cmH{%%Ut`fp5(EG~B~%xr*0$ zfQh`gyp$yp0NG}+;D5|~DIUsp9XV%z9#i_=X^G{_E>QoBAZd-%1}MdN$jddt{Xecz z3dT(zqlmFx5G={@05SIYkeFnce`zUrz~%t3UQd|kEr1*WK%?^dTp2!B3zKCV zD3l~#hwS#t&6tt6%qaNqgwoy*X9H~C-?I@!^8t84FLR8S0{v{ccHe0`7K3&= zm_t0r+33I#BqNN2Uh@2;gJH+Rik+ibrcogdLRV> z7K|;sRu1JtYCZLuXLDHSwOWA6OwN(q-6TmBV8UB)Pn$nr^a)v^j{x>LCgF8LG3?oP zniDih+wkZ1NSZ1DzED>d!l4C9*Q5M9>0N}KO)GAMKqXUEM+egRu*K>QEEunL-VnE6 z>o6b~uJXNP+-qU1YE!#A|CPkZ?zAdsKVL1j*kN;bvG)S3VgI~8abOmO&kh2nq6L1_ z&9jNay)7sxm~2#17I}2djs~`xOFTHPw0u^}8i*zg8%m&+co=XK6AN%P++lkmJWA4% zNlZ+9St8#I%nv&M9iqK8qW%*5si^w<10^tjwj#=lUs_KCFAX0#) z*6@`PsZXN{-t{>OHkq<=!Ap(P&`3YJ^SweDRP0(bQB+=Va& zmLK^tpXO(H6mxO=0cvvZUbkk&DWJ1{dvZB%`msAfeLjzWeF(6c*6( zC=NaL(SZMZ(P-WGu|83CdjCCFLY4;;WLk8_n=b`J&z^ zDTKxhi~|SSBw>;LJ43&+Yicm}_V(1W0JK#o@R-u2V`Lx{%vTWJeL@0){M^xuHI?TV z7KH20XCW2)TVPhNyEh!W;5;HCBA=)4(v*?*jHZ)L{0JRZa-6JD6=nOh>g%opfGJXo zB^42_4wQ&ew_l^u5`>7)K`|JqfZ=&mmdV0WLJQ(!iz?L2>t*{I`#fcAw%c`X}$ zAnWAf!l-4P$#}Nd*BU;SmR*XH^+uhHNp?09{E16!j>nJ_m1C9ej3mRw1Aa+*3D{mL2z~;YrWnRDlLoG}B$F0Hhx=Rt~hWhAb0x-wr zjSdR#fXv$QROU3h)*H+)h@R+c#L=oLSxgBsJs?MN}wKCku5lo(63$ ztgG(mjsBkpG%-zwbBvzv=G5L1Kj&R#d|CYoj1Z_A8v5c_?hXUoUFcpFRVKXvT5?E-VvZQVW zuyYZ0qXG)HcK4Q@X zRx0oG@^p1gX$KDPNe!41ZTmJLwF8`>IpE#JLle4LRI}^43Y;9eB7o+sC7$S1kn*$I zR}k15+gZ0lWme008z9jAvL&fochkhZKQD&~7Ch~OReMBiH)MTem}i3?x1RJrKb<_w z&p~)E9_Q}0VIXs)-lt-F(NEXFEnp*b8Kcee{#!kFM<5Lm{R43Sd|d=HiU7aXA$kw> zeQLpSYQOwe@9-5!UX}!Wfa@Rdsm}xI?b3F)U)FrJXcv?@H|q9$Ga%x8P|+km7db60G!^F43Gt)j$jh#k^c1WHKNO0SXw-P_N$Ki-%p1qCyD{ZSSnnzc^z$_ zF$13wO33dt;wx<;`=S2`ulwS0lEDSOi{o^Qwe^6ez?CuRVI;ul#) zRtijMh$p`361|J-8%#-ZM?XGtaa7gwZc=GT3vO;l$TJ2^B$8h5S(V?T0s{~6;l#r6 z*#PMPKqtavz=GDxqxd9I{{F10%NXCq+$3x%vSh4EtX&%xy~!Ts%H@v-kSX|=#(({Q z_v#hi-3u>3CDwNoyxC=b903H(c8)(%U3_1d7LTen&Jut8FTpOK@Vy4kNYqa}Er#S6({N($x(QLPVEm zh4B9F`zB>dNQp^vV}}Rf?JrWwAaKPeZ}hzDA$=li_+c|eV{?#K8B<%!kutK~<4e8I z4)JvryldeQpKip^Cl8k>zi1cTU7CV3Oiur>gTwZ!x!YlSh#z0ok@e@2-8m z(sm+OYI1D&8*>jp`V^GeIBF8S+XIl?F&nUg-uS_|m}I7>6Imyl^iG7g3me~|+NtcR zbY%#w@8X@J0lkU=(Ej4bKL_47S63w}44&Tntib!uRRJO`uo6M*`+ug<0EI1TJxDBo z4(tvs6LLSC!ZMG5&VHcA4B;GP2{RBa52a%q=&_3FK2atnxLOL?N2qGQA^-qb0ww%4 zp6e4bq*A7)6eJ#hFa@vX%_a{}@mLZg_(&Zrg6R;LQFOkPhmkc3U`c(~WWe#gUs@BN zj=+#nUi~eihELuNTQn8^@Zim4*rsE<)HxDg96%0=y}R=?x3GX8G<4s9PX~gE6cImP zW6fZlpXB@65at2_+-h7#YTxs;HCb+ohcXID35tAK1qD=30)eF@z;|BkPE1}xS4~6u z0r18t0Ds*82K12XhuJ8sP1Dc*3M8hapKdqdDDMECAy`^eS2sd+PtV(TGjW`2Pz$h| z#V@V`XR24}B@N9M#Q=&2CFO|T!1~y zspXR__r7cMqnN60gnDTH#?dgXN*E9gZ~}M4g=d3^tlQ5jq z7uMA|N7DPK(A1Oi4dILD8!&^WH(GjLz9=G07LW&&AbjhW78LrzRViZl)Ii>n)&+1N z$ipW}9YJEkkccIT;W0@ddDYb_dO|=W5kp@v!)?(7S!X(SoAtWazUkbOX?a49(tzlJ z&3rIUR1tD~OxsT;o}42Q)@_XBLmb!AA~0kPaKVX`B~R!FxpyAVe70!bYkxh+L;|NN zf4XGT*Gf2`3`4%|Xe-(C1na&x8Q$zDPp%)R5?@+Y2ZC6PPaw{mh0-_ zF5OtuOLT!{FV}xQW-7qwqVlDt<#5lDbBcvq?Zk8+VjlIyS*)D#9emjFM9;;)c)VUC z#f7iQn7X?>2(s0mu0C~o1W&^Y(wl=u+;~TQKMP&)@NW!}yrMpk4gUS@b>BDtxvjv` zMj{EddYz9eR8+LZW_JEWFG-OZ@Hj#=oACeAA~JR06GsNVTO=q4Bc(G3gznRbs+ENm z^XGjiR*t9z;Uf*1IFi99Av6?Lk}OW5+5@a&B7eNFKc~{`cgf^GutMk2h!yB66Dce93{fNM%ytcR~(r#K&_@3kP?2D z@tfEegG7W3U}ExywpBK@*P}^)o!!)~RNsNs=zjA0L9cZo{qd2!mW$#@q$C0k_?4N0 zWCUS9g@G4O3@6N5(x(YjA%vNUGv7Nd^@at3#6>lYUVG&sR~ZRoev-DpeyItFiUR(T z6yd)&f&(17h9iLp5k3&Z-Omi3#HbPRvlhBw$=mUQ0z?-aMjmEA7gI5mtC||>hcI=ZtIXKRG6)H1Y#-_el z+eZADa?9W7>$`x!ogD~9C=h7kB>+bS)Tv&4H7OiQZ;pR4{n>o7S9L)xa(_9Kd%{$X z#wQUdpEuZ?*Su=!wM8BBw-$k@5#^sfbz!wM!H1UN#JNCTVEDlx;vUyY%uN0#{q29E%#>UuAOrOJ4#Gh4zD?qZlw7^*(;l`H!v-^WUB_=pM@wFP6CtRZj z_hw1ewaHIqi@~9hU?KL`dU@=g2wj=|QT|h=V&Zm`wxtIy8#N*B#3El%M_SXW+ecB9ElGeDCs&^6dsIGs{hZRO?p-EW!%llag#3Lo&o&|=K;w9U zNANk{ocT}VNP$m(zoT9SgC1$d_npmFOJQ&3t_!L?a6~^6Xq4=!&?P<2(ysDd!FwCB z3R+^1FC(>2h8bn41i!OF6VHt)b2at5;ulA8Wd?S|A)d8TFU5ULf>D-c$ySMr0M$^h zpNq|?L>IOo2>M*%FS|m!<_Sn;fI3026c8Zj=q%9vi& zX1&fKX>e%Be$r<)UH@AqEsW7Hl9i@9gW z6j9sUpV;|qWToH7Bm@D~(r#Kn6TKS&_{YoJpkM?()vmS?!sn_PYa1{c8feGd5;BZK zPeqecQJUD{34+KO26~iaq`NE{#2st`d+epZDupP};I6mO!?FoRdi|8KL~zGL=?(2_ z11M_;#@xD7QbF2!JQ{iq_GnRg5D%ExDP%=6W~OZ#K>;hc;}lh*bF=MNKr8YTs7fQn zGU~rnr9}e1EMTK2C%7HBO^rZuRqhZor6O*8f z3V5DR31HE5UKaGZ_Y7UZfr;bG{uMxW=R&`Jb>}?x73;yLkK_-|c6Y+J*CFya$+$%I z5Afb3=o-K>c!sk63eGSnkl~3{m_K6QjuCmu!rTJyCGKRo+UlQ@j=uyBLg&y0z;GZ2 zIB1Tfbk?AtHffmoNia@Ms+-JE%X8Bt+TxPOgoX7TxJgX=v+?OoSaam@QDW}Ij`UGu z|7D|zZIFg;yj{nq$B+y7Mf2|X(Hargfk{T%m&JQ;L8W@|GgXoQwLl=yYY*zyhz$K@ zo6c+U>gn%r{GhU2Hqgiv4G;!|vdc+M_@CAI4hSr6NV1v`3~>wCw|pnOY8Z5zR$61} zBox&TSB#agE-(A1G~7CCS?Lk4jFjt2%Uw_3xMrUHvn&nE+pI{Vs8KOTjy2ctn!Rmj zIx>ZTjD`yL^@NO@+E%vuDOkeuJpVc^WH2mjtt5xhiW?o z)BZ#}d$~yK>)(95_$5SNfG-JRWqM)uWS`C!x zoR7ZtrIYd%dy_u&R)yBNKr8=Y^GMjSm3VO*K|^bLeR4?3{o=v|Kh-9R7a=AYh)1&l zql096uJupJFth+UnrvBY0?(=qVu{6DBWBx36Qc31Yl2DpHF7Er11!gOtROKo0C7=d zk6R<*OT)wt!voFjf}S@GQwR!W@bWiozM>44uZOyZB&J-T*}&$ZyR!Nnuie2jGT%ENal&5(5BlG zb9uI3$Y~#=lE7JqsrMSrLeX)1rUJB~ zz-K-r5tAb1^@$+1_00D~b2p?)(2lpt9mEv8ce7J^l51ikkUM*$Y-4TfF$;6EC)1z3sqq!g$;vms33@MK_+S2C;T!X4gBA zIuy3Hq%$%~0B2>7d_?TwA>STBztqWU08TNY?s6<@sw9-~{!K?xP*hx*)4mi#i(He7 z<%gFwAqSF~(@5C)2VwXuV%kC#vS-NcVn#4zUn2<@({Z#$px>qYLM)gvx)ZFOU-e$akF@ostQv^Qz4xAAMzaA*=m5U(%fiPB zwM^u5h{m84s`;*0)&D=Gr46ZaBH3eU&y*#5ssnnjI#Ip6Yj&^9+84;~Gc}{AMtByf zd3vjvA5t@a;bXqrhrQb;uG4DXnZi;RO_5JocJVB8#i-wgFj~wh=Z0ir1>J@ke86$E zHnZGg>%OK;tJu77Nb^OhO$Q14#VcOlDJ2GBovE^AznTL$<`0nk-p^qEhz?h|jj6Ri z5r`421MRg%8yE)4M`5@-juQT1nSk3#c_vxICx4 zJ~r1uoga-A(dWia+<#!$qq9Gk+LkBS>6RDcX>IPhq{{@fG>(DNj6I)ADT(~ z0z{2B0OiElz)reQa6V8UzWBpEA0eYD%;{wYTyPL-E#3pI9MRBKcS|m)jgwKBCC2$x ziq$2R-lg8{(;s;C5E(l`mQ_?$Ha26aTi7J2*zSn@y4`|dV(DCWs|U2jm*VtbMV<5F zZx)7E9GLN`k0aECFXe^aGK6849C%g9y|fcvu`XW*OZ2o!|NtPp|u zy_;#)U z0H3*Jt{VA#amfNb-2VJ*Lj0;K^UAIT!WF&sxoYXSo1Z&Zs1Xk4h z71J*a99D4PnC*;)zA0E}{-8lP;N;=cv9g4^gG)#3jM~UI{f^xdKM;sj{J3-O<4#>; z3fmnOH%1A*u9Mrg$iY{>=`WT%84Bu0&8)2c`JCI>7sMb-gVjPJS0E2AaSS+{_&HW7 z966r7tyWzHd_{fRPBZDBq$i$W%#Rk^0dKhoyd^F#_)<2ka{-(Vf3cX6`p%Hc9sLrL zHh$is$352K?@=Vq*-~#F)B=E3-t9WtXPrcmGO9s>z(QH&qSVgG7@TumYlUI97D?S? z{rOI*3$vkfDd8PoldN}l)|ST^$A)L}|6Go@9DOv!8DBi#H+ybymUi#;m9K*19e%Oq zWnvk{jKog97;>)=RJ&V$(IuLdns=v8A)lN~EqBQB2C4g+7to|NXfpv056nda^pq>? zAeh2>{pRDq#Fk{4+!r+w%@u9Ev<|Y)8~t$Ku?BhQ=FY<{?IQuvWxbQ<$L486)3oks z+T+*f4pP&$buA=$45Fzt`HLC&+I2YuafqM=!z4y42DNbowGm0#m=zOpZO+6I{2xm& zq-p4+GZwMl`e&_>%%W?y5w4V4-FesPa3!>`k>jr_d-efiJ6Ca=AZc4ulLZp zVYUcI^Mr%3Z0o@Gf8_m6^nZo{pveT^m)+g?ox}h40e>6VtR8Ci|7*tIS@hq-jAlUM osqB~Mz^TFCO8U<*Rh;%GqPF)0Cl5S}M>F)0Ck}jpCySw{6=zTr6dOgqg=ch-` zu=k!>Gpl}Ut(om787WZ&SZr7j5D)}$F(Ek+5QroY5HLGvNZ^&QvLbm95KtXcK|vXD zK|w+pJ8L6T3queP%Fp^bI&0!@DEoDFb#(eiXsBTAoaKUp!{l_lMmxF*A$mmMvpx-c zeH8%;%F9m%Pmhi6-rU;(okh3rI~<~3)8gP~={w?j9#kY6lVHv;l?_rBJGo{+_y`p& z84VZm9t#T#!TZ4bEE){C8MGgR3XBuBKMhPTUvQR&|9}+wBa(`wSQJx~2k@TQsF(z3 z6#>W`r?}YKuGqms%6EeT9R9Lx@ew!#$nPlpZ@O2yS4J~BkYRHKv3#*`eYlkgU;7J2 z!l*Vbz(^)1R~p1Q$Mnov|^{<8rD+sl-AfsrFjG`D)rLg3NmD z1PM89`uyyLXsn|%W1w^M@$vb2dH(tNxjD?o=N1FJ+s7KDkJdmv%JNzb$Z|bH6>%d; zNf2sa8yW-*6a@qv*a8Lq1qm_*f%w@50U-x|gMffX2ZBHWzfpkyWHP}1xeJk$0sc=L z%C6mY$dw zmXMH;+s?pgQJTH6`ZF>r8j(9tu}F*3ddKJnJx z)yhHF<*k)H$*)2F8Ar&_Uf<5t#=+FuituGzT|H|@2OeVL7efE@=hr+9T}=N=$;$pe z%K|Qt?&S_011&w>|JKdH)ad`{_HyS}x1a0!MUMMrFisg$7efnGAyZ2rSAkRGWngBe z=l)6OU$_1{(_dW`?G5b&tu28)9eDpMm;ZGB{pQ~tf6`R>FHJ@k)<0(f>nr&%$zw@}s3?E87dUh0$5*kO>{^{hv=wkT)Amkg$}9%4F=7Mt;+O2K+g!W(vR5-VrUY0D?4x z5dY6lMn1sivIfSK_xI<1Z^s9E`{#Y=)@%*=hr<7mdTH=Sk`QJV_Ps46C;eyczgF^Q803%cEjrV}E=SViEj}WuSl^BZ!vfITsd|yU`$o^tNN&vry2l&-dWOvAv z`bnoKS5(qxzk>t!@hN5s7i0b!?3_?WPR+-wy#)-1^|+EaCNh#OP)L}NI$GDx8&yhKTRU?0 zvG3%9$y)ECpAz*w+X1)R1lb=eCIrKL&o3|$fHHac>pEt5hiy!Z=~!DfB~5vaheo9i zr&YTlfLt0EAzn6VZTQx!y#^Pjz9*}$4iO58LT6&yg8(%N%Sc&P*7BT7HbbvplK@rE z*+`0FTE74vZj58Jz;43pi>>^tD+G|QWm~Y`_4D(vS|d>2OwRt?w(<9cc280Y!fAo-%A%M7{5gK1DbN-DdK4pUZ0`fDe zaN$Bf*}eh~Hc+Ta$(R);kMDM(v|zihcyf7J8MHN=7Q8A*Vzb=M$M2|!fE^|l`#Eu2 zq5fEv*J~0o|Oh1bjmRZ?r6!Zw1E$P0MetPxtZ_s?ZR+ti22z$+ZaxB$DV+ z&S{e^=GPljn?N$>&6g{(d@$L$ET38ArlFR=vs2=Z^^8>399MQyIS%jwBEX zG9X=0H2S26&C7CM5|7jn6 zbRxL#?&7pw*Am{3hVo7C@$(s@nfcyH`|~;b!-KnOZf=U$D6>@_cWf*(R}t2G=ed$k zVnh8_fFxFnCG+UWu}T4X4?*y0tA#~7o&a*rFIZRCOCgds-SXvxG!)Be+oLWP_siXs-rl`1_xd z>_EnPjc0hyfXCt1wM8*FINJBjQ$Jkwn$HQMH{!LGRyTbw>2n%`W69%z7o+pHHt@dF zr=quSi6_De*DG|E6mGDTzfa@cS8`nyjPzX$w=Z|5)@$nv6G`uUQJgv{D;AtwvB6nm z^>`0o)@T-I%Z?|KW&8eHXN~QU*5ve5_ww=#tP(-#vSG50$#Q~hH^1`asy@ZZ=sk|Q zP~HHlh;Am#T=r~FofCGSf}~`rBiFY%hs!TaJVClK88eBzki!1D2{50A4 z+1CHTp#LCIF>=)Sq@t3*`Y&+2!SXrrc|2X=`4E+^ZQ+zQA1a4h)$9M`ji_W9Nt;SS z&V5ty_R4f}Sx5K`9ldy+U&XTyHzF>l)z~##*Kl_FGulz}IVTD1t?nk2`soU$r@_2A zy;RPOq!w9IBsUzM9BOYk^Nfb|S3Fp$;cRBdGb|cp+c%=?(c$ZqNS*JxEyb2AW7c;G zQ2YbV^Hf1Y^cPGa$l^RZeQpjO+1Ag~8>dJ?kw5|p$GmR)$e-n^Cw2prl|3X(*4RZN zaebwiTe0FtQgi~!R|TP55f#&a*I1#9p_jE9#tDG`vRsq!j;`B@6-KUlNk*xRZt~Zx z_HP+-!ZWR^3$q)G*QZWi&W-%RY|yf3KvsWsmWyKJnv|$sL`5~Zm2ax`=dGVFvPN<@aC5|Z1cDSi-^qraY@{IVV*1OH1K1Yj z(hRS{PRac(k)!3AzKxHYXo@58xiInGm;{yfRy5i&j1YNzCEFx-IF2QZOI8kUe^|z0 z{oFW_$Asvao~*3XyP9I^xh>_KEsFT1-+>dugW|V%Wn35ri0fH!*}9*8MmZqj68)Av zq=EIW&5H>f`Y8i_%#&HV0>}EcRI@r~q>s1d<*I!s0R(}WWTd&7ZJBMah=fVV5w6eA zO*yu1b-iIq?Qx|>ZDA!c!S6Vh>spZD;VQ*DuM=>k;nU}b*4U1` z()$yj-_EG1>4Ifzf0~O);AMeCGXRrd$K?}6itNcG5obSrXpWQnBEBZ68?qKHFA|RB z3w7N5jr`InLRQ7@jOSH?!IFcND~7do{o%xU!uf5JaQy1#jFNgL6%Pp%T}#yCHJ^|1SoxF4`BXGE9P2j8$em1mo%#;qh5z4^=C`y= zmApxfOVfvGIT?KKk3(d<6J;YanW@r)D`vQf{?`BQ3Wi?18H$Lexx>G}H92*2&$29! zAv-WZbKOY&l(IYnYiLQ++R34q<*>uQOW0Fl#niOo3ldX)Boo+-jzNOZ3R-R2(>+cd zhNi>>dEe~fnM+3bz&V;OO42#Ib3`$7V`f7CeJ{>d_gJv#s{k^UHan*GVNtz|{SB$3 zXh{b!9)F^$l>pEev#C-d$iL{n8;OrNs)%0j+~M19(q|*FIU4H{P!G`<@`x(h^ua;1 z9{gax7@=g}QLz|h)2;>pFv|Q2O#+E0Ey2Q(8hsk+)ow0SNZdR>-{xm~^Qf&Q@&(|z zgXMOy%4HJ+@cN;K1?IfD$yDjQ%n+-rKulK3A&Q14!S+^boEW1~vVq`LAwzc1YOuIL znu);O)8+U3UKmAF=L{|f(#{PFx5bDr9$s4$Z0n=B%C(MIMZc>$A(&2t1sYm;1t0HTtOs%vp|#=N&SxI%JC?biZZO(zVAP zNn`baI<-J4bd8i=pg3c-Y#I!qzXly;z*#(yY7n7;rp4@p^Wj zfR$_sMl3$@$#e>09}~%wzz(0cYC}mBc|$`>)jfz*hwVc>8u-K@?iy}uSs4}_(Itqq zaNt_V8~S_N!C(K9ZOp$-gnaW;_k%!QK>Trpxsy<50gvqQC^i4T6-}v6bap@GsoshKBzH+c!hd zjx-!v;-8?oL`nU&(bETzOy0H3o|?k=vDf8-jcZVnakStu zD1u~W!@h`(;|t^kQeAnoRXe0IR^_LF#?K-`NWDOBt=}FQ2F;}F;&-?`-FA}od zDy6|{-?FF~p=($FMs-$AQGzwobJ=~(Qv#Rod*J-$7JAn7hJ{|&**IvYird7OhPxuw zx4v=)1|~!7T>^s0`U1xaeG=f(iKrJx6P_~47u zysqBv>4RnBpHMOGxxJ7WZbi0bkyZRjym6N?1R9A&ylFA@eat!~Ep7FuL31;6%iv`~ z5|i7zy=&goCgj$F>XgYv?=eE5h}LY(5jA=Af8wKtScnU{`48gS9=|lCCY0Y>yiPI* zRr`<%-W&%DLr?p#u`GMv4mVQ-ZT*ArAE4z%45}=h61(+(zlrzn{?36H2#Tp=8a)43(o70eV^P}T|0ewZq|3i(rAh;R%{G&A zUYvFZT@XwPC0oyJeB>wPwzqu3?<2JqsmO~{}}%kqJ^vd z!^`T&abPCdSTjnBz=SsqDGT3-iVMGfKXUzrWvZaGRM^9V2bgPrGtrWil-zO*iH^VB z**vSFi=TgGUC-w3;l6Ub9n|5?3%!dPtaHA%%m)1k?zg&sXuqfTXQ2C2s5{KSS#cuN za-#Sw31;M#VI&5)_P-evm9pi1KR76>s%Yq+Ylo)zI&+`}mgd93M?*_=vwd^fnQ(BT*f&wU~^~54!z>XutU@?s?F&pXt$LoF?YQ`?P z*HuGjpJk&%hof<0v7u1ElE&MTHMWH7Y+MP%!~>!=ByGOAZ*0G^rXfu<(K_eDlNiyK z%3@Y5J?L@vG$|1+b)8~X%4(>B-bLTtN{HuJ2o7u#nTW*kc)Ta%rbH9G6D4M(gxBpP zg5Jdp%hdIw2UZJp-w`QQaL)NH{b7ec<7ZB9ph_Zo7my2Ou%U>JjP-sCBQ@CAA}+6h zE+kVG8~h+_r3$(imOU^(hCufOv(Bm~7lrxII&!V_&dy}cn|(XkzCF1p;>KV(TxLom z#d+Hz{u9!->*u2$}@Nr5H3Kr9>)d5P7mcV{p@J%pb?7cvJud^SlCQE$tG zyL+n01?_K9)i0g4NdwwaDh0bHIaq5rzo0$|B`uzmB!3|TjY0h1Gu9)D4N ziCG&V{{X#*iq8r32C)ODIege$mX(vA|I~Z_NaBqALj2m4q%dd)Zl8r{fdlCXQ8_nG zl=JU4^i24w0US23!K32KVf^4M1iZ`d4f;~)sB0a2wTyy~Vx%aKJB()JWi3JL@-g(} zA7~+xAEaWkdUXyu09f26gn5xzK?$AR^FLbl{%wv}K%I8L0UD)y7{8(=!Wxsj4J8LC z%$A3j8?M|L)A)v!cf2jkEj&G_8TpKyz3md;v_w3wj;1uIQSjC#qC<939d0w@mF|ZJ zD}P9YjtgB>dR{vS5d=Il;e1e^$EFK+X7<1Jb_e#wdj%+tV+m!1BJ0Q@yAYMrNs@5h48LL>-a~gVZz?w<>QCQ^oi#&H%V$ zTs|V}T6Rz_o%!W3k@^vbT5=^j23wN(^78vH>jC7bYRX#bfe4@GE7=a|uSoGa4;W%>Ha8N3v5^s#t*bm!1HK^s5!3)lS-p_D zzB9jSjRFzeaKhyb4*%5rIUyD7*^~DK&OZ6!gFg}GM57hVo|qL0YvZuhM@4eysX+fj z_9#G*AH^Dj@dKI$mt)ynYuRyQ(m2PTJOhRv-+&r%xwAR70gQ@$vR_kF9G-ZV@Kk|A zO{}e;ZV05?Si3YP=8EU?Kv1&VUh`-Z#M`oe(V6-d&Yti(vZ2P{O#(CCxn5pJUn!r~ ze7Az*AK5H08ioYrfw>HGoO~m+QUY-{6jHe<_GUUsLxKzCI(xJkvMm$-b4-VWl8Gg^ z?kkFQn=_`{5<7Fd)4j6m@Lf6>}XGDXyXApSo^a-Xz!Kr|dv5gieL z;1$UzEQGm_UaxSwR@6N5?2ZlpEULESGx8F;NMN z;Q$S@5U>(m$Mw!7eMO7OQ7S1G-5c%_Sn-3o6f3M8l}YYNQF3V^US%HOKwwni^aA7w zN48%0O+fz*Q5k+OL5og?nG^(P42a<7_p|EL@4e_ZJL9Yy%G=!(v#;nL4V}eHqavv6QSN8#VrPrR7Zz($zSE=wi}wael^{4+4xE`T?OH177STELRqsvsP=RuI@I6?P zwpvQ^$gKyHY%QJOt)P`SAa1mIaKjST+E~BnU$Ced3Dk_*f1KoWvklE|3)Lz6#x>P~#5yryOmZy#}#c*y>42}7SA z=S)CH!uVyqn?zl9dE9$U`<^NcngBlj!$4irVVBv;9~>PIebIuiq2-0n1Ylp$4P87B z?ON*e(ThB&R}go2^=HKR61|6z3a*PP_2d;L5!z~QXu~3KA?thcqyzJ@{}>CH%D&hZ zZ7nW#7_@r||2tKfwC5e3md1T>g?ptg^zn<+_nhAkVO(xmGXa;gSna5nj+ZVEarekf zF6OK?_-qu%3XPf*ia4jxmtq|iSH<41alLWAfk@@9BoVwDti=B8^^ z>g*8)7jsXHq{u%>EZQ@S=}*!URFi0W(OJBo4OniwwcbF(OcAAN`nB!xTeX}s9muj# z<1&2Vl@y2~GxG|wYmAmk50;Q&FV@p~p*(xW{ZACT7Fn}3tX4QrH%CL1Wp9o@O-!cz zWzTL{Ky~A>c*l}@B$<)F&l=HGe~HsuFhlgMxJUACpdmP2l6BD{Z+^mF0JhMSded^!Nf zZ|`*UsMf`H{hr4;D=G2MQTQWY9G2*tZ1QiY(6~clz2`V_>nd7c8XnlIn@2F!-*Gv8 zL7Z_)u1Y)1la*H&HTSu)#A}KE_SKjHK<0${Du##f-$UnMZmcMh5Ffnf^(KFI`y_GL zJwszAZ$WKA*#Wm7uYX=StHod}DfgEK-2;HB*W#UlTY%uN9ZFV(PyTSn5RsCq5So0!+wd z3>GLQo;=vO@I8q{4CKzN7qKmgbb7>&LFmwupO@xZ5$D&CYBs5-{wSXT9hHyN%jTp! zeK9J4J(b^4^R3e1ysu(q@+vGd*RX54r`GbPjz(yD|55aD|DG(gH}iLZXgj7+Ma}KM zR8J^l;$<6_d2&w-A_<0>g#~DYc+Xnr_(d9nYP|lS-53`C#mp8mWn(@$C z6c4QgG5?h(i@xN^--M>ppxwbh$w9n~Mw0s}$9$car3*?Xh;0zTXk*F?hl}BhudLr$ ze6O^hoTU0!XoU-8FlSJ;8^8wx%=%a(bVT&J%NpdIk?P(V4$A{L6Y|LG-F*QwRnDd5 zRsfZ&>i<3y6yQugGf%|2HxRdmNU2)rwZ^6@o zq*%*h+3;TRZ^o>w0rwRgY#WLc0VZEMZv+VE!l{SKWWfOtR-!11%kw#kWdkD_gQM?(vo1sMT)V-kFMoyEg_gO$qM3$fTtqpi>kTi z3L6*at^WlJ#5Ui63esd^GF1W1j0&tFbnlzOs8vZblY~%%#vOa#J*p2Sd#nw+xG?QDz- zT6p*mpE9PoV!A(+duIgV;$}WsS=+w707B_t$v^DOV%>;wkrL`g*s!#hPpZRE0|?NL zH`h(qXuQ|9+?>RJ`cs?FFH#WGnX))0k4i}n;ORrk?$yj)kZ8SEYRIX+X5bpfV=l{^ zMYWB0z}?gMlYJT2FI!|Er`4Dag}?~#W_>2t;9YK$q&AKAgwFN`W4wp@;KF$wU91kd z(t<8k|LhFpJ@yOl5lXR-N+Ik%zydx#Ko5?1Neq?{gT+UCOY_kkaN8ln%$dg9VeP5^ z**PQrh4KdswTT06F!gmr!b-xCBj#-~Hu-HIW$99sjN-!1`3*C*XRYFG$A1Z(^QANx zkyBXb#^bZsq;|mL1mXJSc0|16BM$%+n3=l^v{cxglLK+wAz;-6{x8ktNdWf7Z)r$U z+eT%NP4f0{QDDAh?)bo`nc5DrSE&d_%a`%v;reN5q%>{j-5=rts_YIWm>b#=Wwe-jC1DPF{s7{M>r^UC)2Kq=sBNg+Lxd605HBdcd{@!6gA29w&I}4PBxCA` zx@7Xm{xBo2=r5ifzMvUTje;_S8c|0@;iN%2$ z_=L&ta|X8EAYOtez1J+CK8O>R*FcNA?wZ)KQCl3z(fZ##8G!X#ZZnoOf8Vpi75{Lz zaaZ)OK;`hIfSlT)pb0Tb8kG|N$?x^4&21tRY%&*WWZSVGM9Xmc9Xe&-9Z7T3Y(F#Z z-#q6Ba!p!Acw{virJv@F(0i>w#}JM0J_Ysw6FAWGe9te+EQl23HMw%e(eMuT#~SYH?~O#=Hq*m%+({V{6%xw80ni^c@8y~07#ligzcVlROVW!zztC+UR|n3g1E z5v_>3QI+?XIr{fEBBn8&cVD9%BZJCIhQ@OM8O>oCw+X72kCY8$eegr+ZKb+>`w1qS1uJf^u zi^OzOH_`)+k3WK+?goudPI5WzB(vROb61_fJe;(alVlUaJb8Ll=&Z3cSPr&AAIu|p zE_niz4q;V823A~cKcwsSptUNnq2siG^uC49${l^1AF;>CnX_ht}OV5V^KiWv?_L{-;I)LnS1jEgxp)HWAxI^GkZ* z*}aDaU5B-;6*LaP=0;goJwt^jpa+AwwH2d%FJzMLgT-V5CL$UZl%CF~we$dg7TPpp zr9-P9nMm(|orjPXP?aPXI~|))qo{Rae^1okPC$;7?6V3kx_V6Tgb^fneRXr+PuD>Z zNq1KVh`*Wda)gWqwUZ7OTWZ5XN)43b*BXON_G? z2`z?f>CBco*jK%hnAnr$edVF#f9;CdoB{>i1We!rkw|4Mol-u$?f0#>zW<)$@f1?7 zXRu@9j#Ur#taXj4Fs0Zv!>&$_Q@PsqO1mSgGSrsYk&pL{TY0i} zvW{ss+qxODZHA$PJKGQ9WXkh+O`c}P>WS0~k82#pZ-u6azBXs+ zC89S7Q19EcBjTinL>j&&7Pv>O@m(s5Ed|yB?hvMWdMFeD*vwFPB&dY=p ztwK@(tpHZZLU=O4iBB2>v^!6pNGzHgH|K6nPV6kW>hMJ#O!nhN zC=KGR%EwY;*Bw`gP1=}mmaiir+K7B~ayD4b%;_A1deX>gdGidhnywWyYwll)JI zO$c^K3`Azl1_f1MYC8TcHab0kym?(?UQ^PLJm*bUFejT-0|J!16A2m5QV3Y?7c!N) z!&equRfwuzck5zKDk$BLLi$RFKIzJ#+2%2ibomz)-W(=9cRn>7lK07LFVMRdqRUa6 z4Kh%gQ~=lq$8~y2cs;;IrY~*9`b{vjx5733izya@YogI)f?u<^gS9zN(9xT%sN7%! zLT`P^5mK!L7oJ*Asd9`clI$58YzOn{oiEUL_EAlQDs;HpcMF{sA+4;fZ*K2=j`FZc ztftte^_4E%B|_WanM8-XS5}CWg!FF~5vq@SjJLf8hDl(jE)|V5+ORNLUh8)HoeWMJ zN2S?=p=KMeJOxDRNnHpuNG=d}Phd;n7wGDeoP5DN5_J`4AdhT>f3{tWM8@NUU81$7 z$&zHY$3ejkSFV3Tox3${u-tlVa9--hnq}+Spf2)jcLpIxNVEzl)P8d>eUtp4%BOLM zLj&Ua(KWmCp^pO|2?v3IPTPX}`@s)R)ZPo(b7Tyxi0sB78BI-7X|^QfhL9$^31#*& z{B|suInmFl$ws3rMCJ5wUfSn45cfeTiXHZMHi;*k&{DG|c0nxLZgyo{^B-g8T+4Er;OD+>0Ta3ZY*PFaSnhzlfIUc7(vAm&*`C>F zbvYgTg&N~4ujL1*hqJnT1~M~9PxpBOi?eO!b&(&bRS3-}tgAYEX>nUtNp?_W1YU4! z=aIT|SKFVpqy1j5W2}6>m0mHQaYsV>x~R9nt+lg0G1E3EA#j?{`T)!ERkgvecQFBt zV-fak#JJG5l*$VKgU1j-*oj<_HQiIC?VLpKS-17v0{>GRXoGI=VFRZTLlxZjQ;DnN z5)iZ?4ik@z^@s)#w*4@i}WNx(~*S0!-sX?#!TJ~GR&y@P?Zzn zC{Cd(;p#Pa-$(W|2Wj;+c?_-2P;@jDrwM73q&?bQB1J{J;$}W}*-i{MSGugp8ElZP zhp&094%*J>cNP>^Y|Y-^^plR4v1Bdlv$;dXgqiPcsa~&)AdIc;&pd*3OW>C{U$qBb zLMdlxFN1C^Xg1h>KQxN36)GuIwK^9;fDh@ACwM>+WxUF?UM74vnOW|O*>3b4}CrAiG>;{(V3@jK0Y}+TctG`L^G`M8u4}f%$t)_ z7t=2Q6n*Wkn9wJJvXI73f9c+g3hrfc@#VM%{GjM-LKeCK)2vMon-iI#ssU&@2d)uZ z-WjbiSh7uFN=_rdH1rF zfXu78pbk7#!JJW&gB{IX5;3_u7%_1m7==8ZVO-8SLGrV| zWe{K0XikF9Is@CCo&RO1cYP$%1Mp}8wLyX-C!WMYfUckDYh5GGA~TxP;iP5ae9G=7 zzn1Xo-4NLs-5hZdJlutLLsvF(&tOc}XM|Hzs6%!&sgzTnteFv1zJOOGWi`e-ta>g4 zi!$bGOMOo@nkLK#Dhc*GUs}gjOG}?cl-^O7CALnI_C<7kJMg>)&as5 zzXOe`#cEdP$j-oQvz|FM>*^kpIXnU+P19S`0sQiDiJ5da=j%2lgB_8`r} ze^)V=l(uSnu3uIsFgUy_$iQw;Ac>!<6&EP8&YRWHA@oC8sv$EsnV*FB)FpPnIX3z0 z2~9=@Gn6`F={sS?bEV6|0CA&w#;wIfpSQN=092NNV0ym_CP$k0)FJdsAQnm%85D}uO1zcH3){vi|IqvZA3iW z5XC4_R@4%0!JKfy5cdefb?T=~z{a)oMfYjLymY)9Vqv*qSM-Xz9exkf5Ks9L3@W3t z4fHm*F|S=Dxo@{AAvsDbIz-b{{q)@dmq|gnsEAXQCEa)btH+xwz8_CJa#`TJE%GUJn3uO-U89j#zO-7wI*LSR5+V>rS#$V-khl~M)~y8 zYzXHx9K@)i4)i1#L5%tgj{R@oclj0$ZtBLj@b4Cr2(lOaXJyCrY=I|~S=}QZhv^)4 z6%sba74tk2ZYyQp6fuLv@bo~ zUzhX8Ozrg%Dam)r3fWx4B_88hpVLQ_YYom0oo3okW##F+{qq)?pX6_PhmbNeNHLc^ z4nH(_-dQ$$Z)oZDkF3AMh-ktdx@rk_c5b(TC^T}8J3QII+-hbV3alE``{*wL6)3DX zT`*L7gBEW$zQ9I{E5015*~hAHas$mx{~VgYU_l!w62`AFO+_&ohq`~d)|CHNv<#9X zHaX?JMe{cji}Iqpu76vw)OzKoq7-UGS0e^n}hb!On{-id(m ziQ3j^5L8RdxcH94MZ_-yQD@i3p`sYPJ}KWN+<83<(UFQIJyV906k?RAx}gQWIYHGC zt~(X7eabJESUaftA@|^VIc9lg0_D{Hl&O;{v|a>rg;*EF^9mQge^6T3%nAh~tOYwe zn_S0a2YP*!surege}A7J3LdWStW_DmKK(X8*NKQoS452Foz^>}b6aWQ;#{MB0!sgl zFWZDu?nSnNo{f@gN3yyM__aFrtlvx>+0D}tG}o9Nvb1E~$4pJDDmr%B_jE?vn5v zLcM|2+FXpC`P-cY z=nFj^v*gDKUJH)k4j>xsPL?l(?2xFYIbSiSP4SYc>SJtD1Q|}R&8X{eU5b7jn60U@% z`4CK>ws-7CPNMqrK@sKw3 zBTFg=#+g2RA1{}-FUy&}F^D((p?T7qya=j_jU&mftNNOFB*GgtNTS;y#(1#WCgRES z$w{=?cg-ROy!gktLo#nrlI9WQ1wyq|4feW~xiauIrc~SaRMPUz4TV*IX1Dys5E~|D z=%I)q9dp3MMnO$+sjPydu6MjzS?@XEW4;%LcJtMJgZpspw9oqCW;o2}n9}S3t{_8< zJR%AXRMBr$z&v0u&aUlb1VRfXv^_f%5*fxC` zesj;(^_-hQDL^r)yYV|yj7?84J?coAJzx8_X$eOY?&H1x>ehg>H<1O7ct3rK_^@JDaI8Crx;EU9FbPU601YJlx zEV5;r%~>VDQ;Vo65yEB5bQF~AUqL#)dCN*s!jkFRU188Aw+~UwV6T5>=@pZ{7f0o+ zdOZ<=nk+$lxQd0t@aVG*Gc+J-%*>ghNk~3h8dO06Pqnh19Z0S zKF*kLX*Gul#!f$Q2pD!Z3dzwD@wSxu-u4l%2i`?~eKuT_G5tD_G2XX$){uu*U>80v z^tqyytmc?MnUb4|Z)s!8Y&3Fenzdl++cyQ%E|`Ds{0Y3&U}Suo+GYScRw<0AXGUi{ z_08PR-_pamC_w5ZT2NQt`3-k_1UN3+I5*10aP4m5J@?#o7njVQj2^{H`0hY!Fg=NI zGe6rVw^No_v+FTACr4F08#`H>*SIDnbQjciZA>Hv6tarS1p>^fLcgm=8_QqZ-EuAW zMjqX?BFu*{G-rGR-8ek!w2RR(?a_@KddiZP7e*uj~YAirCiI;J~cS<71g9f#}5umQfy_gg6Plvl#QS2Ufz zrmyHmoDoFN;n%fKvu0tqSbpmH z3*p6jW%Y+L6|VaoL&03OAWv;iH#;})HF%5|@SD$)r&rPSoE%7o97UDqHkOP>eU`1m zwL|G>Z?Hv@HD9H6$@E~daw8md)!XzC!!vPBC?@9YI#k#mGL2X{J^#4D>@j5CHny9d zWI~FWP%=Nx>>Zh0ZYWveJIMolV^k})yV4gi(QDAI1+4t{?7SO>Pe=%CPEHOX8yhA# zBz#8{k>J5c(J@9A7IZGB1AZ;7w8#2G1hxfl0Ex)#7~W$14|4+0CGccc%dWV$>IVTa z6w4ppqr9!*Y%Y8JB`1NJZjIn^*-u)2ObGMDcq@0Mc`?6w z#Y{L%+NB8Gl)o`}Ie2sPc!qiEbO3dNGmhJIIiQ?c1X%K-5A+w&H`$;hiG^qRlwcl} z6d!zXKx%+)m~ms^f)gVabj{@09i%@THE?oyDyX*w$k|Szzl^ciNiHAQMn&ZS40AXT zU*8l;NqL4qzYeSDfJQY^4!H0|BOy$GNn`i>158+Cw^nZ`G?Pr|RDmMk2;k7|CG&&;JErUwux`iRtW#gUx`F zoz<&f0Vo*^yNOM3MH5%G?SDJS_)Kp{^!dLM`3*q)4S5vM8|VI*e0tyH5Be zuqk;Vxm0~K2JFB<*hv&HyF(J$9pLfkawho!-vq#cQBYQ1pEFMXYCpzf?8ICq5N_Lb ze|M+$`4jOtI1;zFZg*g9Wvz6X$m;4g-!n%9uIZBFb-C9gP23w*FzfTC%I#E?HAzbhG_?SDlhzu77(U)H&pw!3K%nksBU3qKe~s`)t2K=r6k`X zgvn!xxG)O#PT38u=n#kZe$hO}lgXCNIgHC*ln?f!du^@+79%>aD%ZEQ zPOz$Tmo}kNyXm&nb!lL-RsKH#?l=?l!xme6=kn7PTTX5&{@xf~P$V9gk){;i!zvO0 z9es)jHP1xg9SKwQ?Q$yc4ST3{D}|8I(9?(WG5yE8OC=Mxoy;KGr12Wpb0eBNGz^R% zZAXpUX`$52r_1dQEa+&=&%_PG8;H3o4s8L?TU&RWhhJi2HDo?+b447vEfPH6h?0_# zsjgl}a2)~n)ip8Y)-iwwK|1&}%q_b%BZItZYw4bRv`x>rPcOv-x}+P$RXi%+w*p4q z?X$3Pelexn>~Wk@}spAHZRd=EGRaMoU}%M>C5JQix>DZ6`7-7MO`-XH*MoeGZOS zF59gL>@^&Pt?@;km}D!A?cuE!SER!!PSpO@p@@pzzh24S(E)@1oRv()!?Rc?VQacG zo+&0JRp$%Ry|eW3m}mdT?VfV`kIq)-v8}xkI@t5`bM*vSmnM|}0i;(PSAAVx)mp;> z6gb^jyWD!2=LOQ~yxWUpc-ju#9ES{O#{xgNj(r>^;Hs;t4m`GuoqF4_gBjs_`=lNRBVXX&d|00w9d$~5> zePgZBEu5e+E<6%&KdJ(;G$SKpqNo~wyU#h!+Z$w_T$2L|5trQ<`gPxcIUW?Us!a-- zdPdZPIP?yqm;VijnhJD zk0N0#y)xQ-#6Bt7gZM`dB6o)_GIDagla)>^UUzoN5cYsetJX^;4W~K7E9B$K{(iCh zIaTGhn1IO0NXzD$8Z^;J{My3`o`_Y800CT0=N7i!u-E$be5cj+EE~C>C_9+W4 z(y|LAaHFP^1+sB(_iain<{O{!j@+GLUw^CJMk@>L4w;cp$oW7;LD4swxZmS?bvQrM zn4@`o5XqOX0lwsgOLuIYFFn@lk`}P;c*GTw@WA9fV^f-X|3!bKzrW%d{;Tm4-6+Qx zE7w{z@LWko0jnE2DDX97C+c@(Uhnqo=_H9=BsD&^CRmD0^xkKSBvP8y(=$vM_xK3SJ`FDnMmmyIa$<_ z1Te?O#`ffu_?J<(0NcdpNXdI4TPgB`ByZ!PylOfNEXfkGTi(NGJDM4PepNQ8I5|l2 zB`wV!^TBWc;XP0)p2rw3ltp}{XR*oK-dLhuZ|1yzsK6CGgSS3l z=-3pZ5Js?Ezf)34xIR~7UzwLU9c~_x&KOqOc>GeenF)4cXL>_b(sXW+R}k-B^5}%6 z+*G4yGFo)ew&!lY;lN;Vbwt2zS5Gn_P15~0QVWQ6hmgJNz>_jcSh~Nvn5ncOOf-dB zJXFGg+JdR1uQzk1%Dde-j-8sV2HM4aw^U6lB-(vIpFLbv6b)WrAf%_sVf3y#oF3r!b`k&*|E z6L=9dfRhk43LRw`@O2qIG!vhDHCeD0@Tg~NHKhW3_&9`VUT{Rpa--u&6CX^W)#`;( z0#UXHYJ>BBaC3%NzJnE+QxLQjn zF6(g3bn5U-!QoF`*%Af8*4yub&|6BDw%3A>%c{?=kMp;2(Bu!bPl7iP7}# z^|5DE?X+e{k}YCiEJ0ds)yBfFYU{8X-1aJV)Xn#f2Y7o>US4V7%ieMGx`9)a)J0{+ z?I7NGO)LSDDXZY2d0~Nxd@B+?2;!f5Ev?L(RGT}?gAQ8f?fI5k=Vm;y*zd{~EO_x$ zyzcjWybH7Bfqk0gZ-^-O3ob?S+HZ8E7N8L~@IOd582r1ZigpV8GIDaf)SAT%5E z(atF788#06@D*P_wl9>>^s=42%yCI(C%EG45V?dn`oo9C@MW5tZ|>)P*baFb^-njW zvQ9byeKS}K5@1)&`7Ftxw_Q+Doxx=XuXI;W7~vw=TubV*U$;51tc>rbb;y!Nb^vm> z0SR1ryzzX@fA6Gi-kH0cphrOk=jtfS=lIu|iAJm*yA*;sC4ao7*&1^P{1Vwdq+B1_us(V|m#U$y(TOKPJmN9*Bl_H>W- zMS7|IFTog<0NBs;b_ONlkE;fny;)`o?qpr0c~0CjWHFT<67{JPHq9=>_qR*;_Lw5) zYkj>*BnqlO*XQ|TWU0ZITFB)ldB!O8LIgwhr4m}$8RyJru37CbIfwn_;xG=Ir$%9Q zbz)pK#*o{ycwYB=ml+qeM%d1Xu(>(?YMH}=geJ$yQ*iJRo%M<G+Mq6{8ETfVXv37Sky!w_Ndm0g`E$k1gk z3xgx+oSrBOR~M#`f5FM+MwsRhlRh<8XSFL!;dqzja)&G9EyMs)|AXuE!bOu%qCpFl zC#J9BG|(hk4&r%)NdaX7x?i^L>R-0@??}_1U-{png5Oy@q2Y(Z^zBzGK~GG{UtP8i zCD>$qh(X6lW1HuU87700fLXMP%*^k>JX$`P&%1tVGGtQ+^8kiV2HV3tHq0{!SAd$m zf$jbsf!R7>a24r;85vp)pxxys^;qZ`=d0K*RLE;jj=zyC=A7NMKPYo0t$!kAtmh+6 zvxSqb5YirCll3Fa;$auzNc+z*#?^39e1#nE&=Z?PikJ!MYU?2?u5b?%?{ym$C%Cgo z4|zxqfbB!<-jURX8d?w&FeJBz%w)hjqMg84zEPzvy|I?V&xsEFVl1Na{Vx=VIHz|l zDuRark26)<`_Y^=oq&qfiWewN;ujQ5Lo%HaID;}dmb4~7YtD1c!DVr%=d_VabQ8qm zK=6R_05lxp$|)2(LVtINJ%@duATjCzTX~H6qNc-he5^|tCUP-Rq+eN7)+t`WT|$e= z4UJ1>@97xT-Pnk@5T46@-go1ly)W8w1yak9X;Mo4IHL*7_2|w3ET&lX@Shmm?Vflo zTH>p$`3`$JHzjZzhC2H080LlaNZ6b7HU#HB%F46zW<|Tpf+N;rV8OX2o)1r3U#KXB zAOs&dw9SvjdBbfjHh zj5XYsd@p)BADi@G>p+>n=7-BkFg6+Zxcy2vGrsgniK`mk5Vqz1j57a@&!e6`KRO7~ zIYl9C`8jXX>1S8AqkyCp;+{clFdoO#RP2rn3Imn4@MUxSBgE=(Y zY0rV>6opFDgpXvkDKh2Y`WzUV6lOT&9m~0(6l*1ZR2S^g+V+pq8ibV|X^4b?RhF1y zc|p@!;g6DveSRo4zQ#A^h!6|Rkh*iS;mBn>pcx0j_}kUGj?J=VDUJ_XwjYDJ&yN-k z{H8-6zoNMsi}Az2Q`zH0-8+8jL~;DCG> zNr!D3kxG3oM`tYW$Qas2QrJ-C!pZH^tSz6ut$GtxfBCqKku3<|l#Rl7&=l+8wGsoD zWEaP>G;`j_r(OW6PhH;g-HQQ7Ew7g#&m0KYchJ;$E<1AL*6 z|F3!<5c$lXa2%{779#)1m>kHK9X+rzDDUha_?SMQGkuh_3)>j__$gSC4;+S@dnKpJ zjzdd4+;Nk%SH0#9<@3?BX=bT;k?@G#Z@R?TjyKNGZz z9j>fyKYB%x)7#lBrXul-!$T}Yow!~{Z-y|0QzA(WtG=DXZ!B6qHad{#(NE1(Nkc~6 z^B34(^SmPvHuS(bsYf9X@c=-B#QX5+%2HO<_^LX6j;C5inJ0bDI^~b2Bn40VEB7jW z*Xy-svh&-KGA%a?2Su6k>s~cG(lOdmS&4nZYct{pT&0cUc%UMcfS?Od#ZPk1 zbTJ;OMfXPX%|aCS%tiz{MN)0G)2+0pzP6h`g5(Y z3)TfSsqS|1qh^~qnwX9t*DJyHlauwKN_2n{`)~wfArRdY5{q^qP9~#$oW6WrNxRqr zOYFHG)adrq{**n_Szb<07P_0pDO7$N@`&7koovo}hVDS4zE@Ao0cx}xSE7Pf)0okh zm~Eg!jtim%z1}x#X9)7pi^o$m*X2`>}((J-=;t{E;qF=(L zC8h0C-yZAL+$fz8#=-78e*99O1CKsJ0j1y~5Kmp5FIkI7%tLr%51&UXkR$159K9NIgDNfTg~BLtopY7%=mG5_x<4rV2%T+X?8Q+a`%>(nO!A#C17g) zJ<@Y^BpQ*QBc|k@!z`p#wCbVnk>LeRgK3YY?`HC%P&7E-lDDKWjMfRWo4e7*sYTNo z9DctVq;wTepSuG-vEzcDU$oV=#SQWJL_gWj<@dcV$evE@w`{4_770h zPz$-wyo`04&)jR{uYbuwJt3M5kqfyH9snO=xk;vweJM>JMcLgMW;4Z|>V=CpH-NHH7fqN6dPLs3ZDPdL(V==W3Pf zdV|g94Z)GLJ#Psmlv3NSD9D~jX~{7tkU=>Inb&aSZS3--dpPkl#STBj8F3_zW+_ZR zRw59vjd=jB;UBd}gaeB@&K!SGxYvkChls)!Dt$MoB)-%15XETC9~bpp#rb;U8NbF4 zvf89LAjEs7?(h^AL?A0Mv6;F8`>-2yf!PJZ0~uM?Z8sDStfAZ_4{TSYvp237NCTyBP(Ay+r8rPlcaS)ncej5SZO&edXTZUBfO}YR$w! z;l4s-nul79XQ*p2#ZOSx%bzhQC{s!|)jW+LgtKQh$@hjbi?g+MwtFt=w(zg^mH-%Q z>!FB4mpWQt$j{KtR6PN;xDKPzJCBp|W9FG-#RlX(PNXX*@@PPm}%$Cz$u-m*bb z_{+z&QhP&97xM=Guv&YQ*nFR6UH#)RuU5FlB6YnPN1fEgybJe6>dvqad)PS{o*kk) zx>B7Oj~v`PTE2qA0vnxNbQ^iIgu8Q6)WKJom*MyO`~c^Jk?gwDibtY%PoLnbr?H`o zw}jE}SNCz-shmD#&3PVMMjCGE7v!{J=}dS^Rp*O_U>j#qF(|~b=`5ZUiWm4Fz;|<^-sbkD+`oOQt1a+iTlBlN=@&M z6Q*XntX5&8h^pmA?MvrWQ>pG$4WIxgyhdkuYB;wj@E$H+O%Z{_y7DvC9GCgfkphLJl=p1qFm|P{_~3hA_1y z**!~m)Ui$z8v&zPr(qHet>>S0apwYRi@D|TDeebg)7jzInEPT|m}W_SA$_n<(j4+n2EOo=1`o~5M0AW?Ny`_Y6=k(ccw6Pt>9&xI+3=Bi5^1%H=V zkFG~&?&!%hl|$XhkjU0%;4V`?u1%o+LC_MEkJbTdu>PSrF<4yU$!qtHP{K zm^SMRp7Y8WT|HLHgt(-KxD*!*|JA_)wnxYJL`1efrvLc!1ci5(wu(6lNXwE^&(b8J zTdawe+w6A2qla&un#nui(V$|(#LpxqtCMyklC{aG)OZ>|OS|ydDrJ~vqb(W61eNxw zbH`gyQ%f8qk{nJTJW*Gt%NxFGcH!~yl(0`syeY^_uRujzIzVncJ)5oZ?;E#p4d*C% z1w*_pv14~UT1l`$?2s2C-`TCzW!OE;z2s+}OV|a6vEiVj;fI!T!M&DTUg zAkicAK>t@5^5oAs(6@-j<3Mh4pqdK-$%gxzro+&o-4v_JS%Ec4zFE5L>T5o~U z`uS;rK+`z)MbA7|fg|BTncaL2JF^qQk-69~BUu5sUGkkK5V14Mvp7_xOV*Yrr;H$Q zMx_+8#g>p;75TW9k|^Y{AvzK*F8-6%YbUZ!OPu_r`rkIN@{%#SXKd}r!ylru6c@WC z=7dB5Ktwt7ITDSSlrlP2-;y6PFAw}4)5U1 zN8Wwz;ghgI7pVL=X4bBn!)!@JP8zd118l56$CNr5v$S=owrKlMd84?_Ayv8J9JhEL z^|jvpww>N`eDs}|*S5UW#lB@YzkQAxm_RAFVF^hovrq>liM|APw9oNL%_%-Bw(ZKe z_pW_$71^2x@T?n|4^_)O5j)fXGfSlC?OB+{x6nnZdi+}z~H_?RbnJLEUS9Xp> zI%1t!3IB)g1?M9=Qt~Ti6em*CHXEnYeq7%>)Uojj>$k{MqPcN$I)(Pc46pX|Wp0wi zgDz4s#q|`ko5xNX*B|S{w3z2%X=gbU3+)_4l8+-UOue?gPhFJ+V$3;Fd*(`NS!E@f zJ<|5kcBGe#-tdJ1XpNcEW~N=Ijcx38N75E3q~)IlTk^jOLC5LBqQaCUGdd`6PAwVZ zhvecY`JO*`2bseP}`&-!^~K>A`n zNQK^a(m(-fhi!qRgWH7{WC1^OG;39@NF0k#Pl@DORuIJ7N4-Q)RXz|?QkDD|g-*01 zGppo+IdSBG_)Qcx(Q+&o+0<)ePoPsYvpWUNjH$mQ60q&%1=IsF7Yay}w+Q*`#b z{-F7$$l_A%;kV8H1LHY*zk{oW7q-V2q@N*vD*pAazFc7fNNGdV-Pzljf=u7NR2`Q0 z`9PdJmrV2MExRK73QX|cwwM&6)uY&*B%$-d*4D1|bG4({Ld~1!FV#C%h8$NyerYT> zxjJT+ly_U`O&dg>cB-+k=jzLoNM^xL-u1nlZ`mZMd}KrlOLro+CddfV7|>aX{>7V| ztie^Rfth}J>3HeT?UX98_TlL@oYT&SjP1Ve(zmhZR9vW5i0LU44fT^LCT9M*No%UW z>hC`^9bcXQmYmbl(GI-Ml~(GX0}T~e>sCe-Lgh@(oj!1z*qxCN-c!70W8v8i zQFUh2Q1z!@21N&bTt-Ak-A;R&QBtIA9^TbN`Ai;=YnM#Ktf?0k@lGWQel_IiXX*M( zR?jl5{n71KvPe(8L=SZ-5> zjQ*)m#P`ZChZ0gP8EOvhQ1w-pzB{}_)wFlwXI_?nE`3qLrKHw8ybD$Bgq(7s1cuhg|XU^iBU zu6#JdsU25m%qjQ#@tH~%bVoJg()R=8V0`IENqztO3Elfo`lcz1sNhxWjGP!nVc&Slevn7#11$cK+(C6zuFb|Tti37s!ksw_d0HlZXw1&HnKkW z#U*5~r+2$kewgr$ndx7)CAqKg>Pp9`G~Wq?=He?PUh&T3NYw~e{aRMg)-5P(p1;oO z_j;+;#r2Xt#F6&i7r9^2b9ggwAE0V&YX8fiAJi`@-LZFYsIk*71`@p*2%*I3Md*dpJP!ozg+3z zwX%7DaCGCw%5AD&?zeC7)i{lsnL}~kOb?}Uf5eP+p1|zL)mPzc0;}d!F6LJpB4Zq{ zG!>V{7Mnd~ZB{2ln~xqx3QxmUsJigzmb`X~>sFv#MKZJGAp(YCC;L71eaMMlj_1qv zi+h*zTd4~rc93)Nn;-6TCAWM6u#mx@3e=Hi+87of9=s?cNNyEvcr1U;!Y=x}kqVwD zDJkhx^2a89?Cl#=oyep*Rt&2}cGp1dJW-9Ln)$N_?1VrBdCH-0lWQqds^`2ucXDrmuapoDmALS9`v+IGa?>?wY#+h^uhyp=S|}P| zeF*F6bfQh$I6rb?>redlJgi%h=uwRPj(>5a1SUS{V#&Sddll&DmE_iu-J>R%!=YFn z$SHr$qpQ}lLSS5krhrIs%KHrhDsvR_L4yF4KmonTP$oiVuglQHSQ;UymU?@L>)0|q z9tt1j6jU8aezQ*Pdfgw+r2Kit@pwzn{o3LbJEy>de&Sw?UD;Smvyxe7(@|X=wuY+I zoY1S{l=+_c<$$bxPDa4oWFFQAbO+1x9z!D6eL@_^R#imWr4(;n8I+qvmwYN&!F+F` zZZ-4GUXyf5Os!{W*&do;r(4o8D1URlJN@di;ZnkxCqTJ>7Y0F^J@B~0mS${~2X ze{Rpzotw{*GrV22=iYs+33`^}5Xm>AuZ^y9NG+b7Drvn~e`^mr*gX#&1qw4MWRJ?K5lAkBW_Jtd`#*PYkuQ8n4E zy~3zUa7F!^&(-gADbI>u^uFv6KUrs*+^aF!IdW+K`A*1(59zy6WA2Nk-gZPMgf|EE z>c%pW2@wMt3 zYvZ|q<^v^Xwl1|R38sET?{YUU7OjuBlcvvp@>>)xglimp@OZ4@ zdnh_Nxd+HB&VSF>EOW#~FW{QX7Loy_~8FgCbd*+~k7Q6)lCVJ3!u^WaTv&tWW zuJ5AcaW<43+%O_Ux$y9SIJ-M)SJ_T349YsG?7rU_Y*hqe(YnZUd-ewEu#B{?`(0V{ zO6=sE`{p=U#3FEkiXUQo8=`Nij%7wRfI{_9s&nbn=&ruBs8T^ZNguFq?&a-yZAAPLaW*&~yvKCZ@-Y}5P7j73HQ?e4X5NgVq`Rch< zFmbee_FklG_JHhZSSdN|S?rMdo7J7z>C|rc%xUCON)3a`RLzmAo>eeAo}*9NiLe7C zV-{U|+S|9ugu~!y*~$uGmzKJJFeHu(YRbwLVboyYk?7vli;s+{DZ;@e`%Y>ID*pC9 zY9Q%DaPH#m3`O!Q|I<2rsis}6rwi9IR8)4csmY;5ovF8X&aakUg(a^)!3S}by;`k# z^P#4WH_@4_S?c8FheEwDS3@28QG;DT+BsSJZrZk+j=gMVGy1u8&f51*Rz^`_L9;xD zp1*3nJ|u8zZ78|V>DQwn*`@sBv1w(9_c-rTqM*e0(rA|)F*r9-$0pz}ceeE+kVYJ0cb2Fsw?sS$+@#<}GodtozUagfY<< zH&T)PH5=4Fl^S~zYyMtjOg;T08C^;Jba}(~t^>s0S?Nku!N&5q=ThgKr{$-2Q11-B zJ0X0*m5B_prMGTbs3gd_*q&QwDF3zg#a30)Mo!~fa*>1*1rMuW*6iw}-s0{4(Qw-p zm`gq&5({?d`VWpO`P2vQR~1V43Be8-Et&oFI`K=lsDtI%!?hvOCJ5-4YT-|H1dAEnW>NN-=}| zS%u!Nw*J~yRtw`yq*yi3&eJ+o-FZ1`^`2+VnsjBto=Yu%f3XX2fql28k=_>0Xo>1O z#jW#LpN`Tah5e4Fn!n#SuG{WCfKNu)EbqCN>Z6mX@(9q5)NEhBlxjGo?|sqN5fwO_ z_(`&BG75@e$9cOJBx|)r6))o+LPMt7Z*g>|+11ubS+ zmv_oy_qnklANYCf8|>}Mg&)$(qv`5_N39RGoLrgOc8L;Ga>H}Yj4Xy;{4W&@dW4)K za}bsVOwHLMIhk9o%Hz6 zfg!+quks!qu65#s4AwpzORm*g)CrBxwNCZDkSYruzbh)r#&Eyg=36Da%Yb>PM{X?HsN`?yc~=bUo|8jaC;}ar+$&UTE@Jv4m2$L4}h< zfRI4iH@TS}E@`d@OQtu{>H0{%C)5Md_5*fqEAIDK5v(uueW}K_u42+#!weUde2ma* z_I_L4IoWn?g{3-@azZxK8l>%muKcr&>aKYC`4@3Mn68FkpQH+l{?!S4&(K=0EV}5C zQ$0Z=z2ahe_7&M?sXPWusuwR(eLJoDvCwmsuo)|xPz_{x^jo=yb6sjT0z2jv!O6OiL381D6OI0I9C;Y86)%m|!RjUSWSvT=g zr#tuV?uZui=~r};=w@&7x98e}XlrC$EH}b!frn~YbGo!S&FM6DO68fNAUmH!aFEpU z1{|#b8*r$^Ak2u1I96sDq@`L?-rM6rzCAi9|KWx`oa9SMY~iKr9~-D`XLw?=^)4+} zhe#klEGo#(TeC~%{z~(_?4qz3KlVF0CLEi2wRBbkmlly6 z{+=Z9#_|{%=nne_rJgBCW}THRa#jmZ)ve7YLhZKUm5;Qh*~-4%qSEY*i&e4n()i!twWbJvB?fUBKu*P@(ZzEm=}a>fZHxDm(oIOMZ>&}ZXHkJ~o39EmJ(edRmeI3H zqr**v@jtJUvAdeSahy;Mb04}!qG(zEyVax>jYM;M9*tT1Az(&

OhnGX1L!>1*qV z>@p$_7X^UzSeOQ4k2F%#W_%>D>+bZou{FC?ZNVAv$(WiqUI0657iu<@s9rp(4g5cC z+5hw9@FA)>8)*hul{X{C5j9VXz&(2YLQUSq@F5dIo>{K!XEneFBd*-xH3y~W>7#>i zMQiJNynwLrcDoW3lHl(T-vbPJMB32<|4E{(7Yu^pLz8us1i{bKaz~BU7*eZeoq1R# zDC3~YSj($Itbggn{}h|sP~D*K__pkJ{nB9AyhOwt4ND%-sFAN_4{L>c|eu(qESqf4(qiq8eWCCcnll z`6)0wVImf{PPzONYz5}<9d?{jf{78Cx!I^v;DxO*GO|rAW2ODqU?r4T1O)vdl7?Rv zRHN*P@&>7e{GzQk&}hrW>%}EM>MwV>vcNC5DsN|A{QMWk15q!kjT`fLyU)Fakx!zQ z2vHf@@oQD*I(4ZCN^`_E>cF(mG{;m1C&C3SemMOX3sI&cYQ5Xkux-{(TtyXSCdcO@ zws^3} zDw;W>1Gu%sHhM|UWeRNmmec={+0n2+ELV(buB5+F#Nm{ovEo;oA0A4~7u7!{LAz&! zzu%VoV1*A$_sG8&W7YVtACh)9@9@whTuxSjEg|>^VND5A!w9r!4*%;+koc zBEXU`C1kUg!_VXp+5@LQgnWXbtJ4}@kM8_vV@fw}` ztAo$ZDlhYVqs-g?vuPQBHjUl*jP^4|Q8v5-k7yq%pO>91e=H2wh;#U{Bj=`r1*38g zm;K3V`tLpag4#0%V?B`FA!^UU5Z@GH&1d8{uOA*0LH+-;-J2-hJD`MJ zaS{Pvz$_v+n~&XH<8@qTE?REsveDzT=TX9AYGHH*RpL!F|5`Ct6zWf-DHI3DM?bJz zG4dGXeD0R_fH+y+{tDZ-x#Tx$^2j~@H&!;ndft!!Vmf62qy92pTJ@9~c0lWHxi(*Y z<^$U3b;Dv8BAU@pj4v{q(0X4Ev?4jfm_;PmSSW4B|8-PBs0n`)aHr{fMtT40eSAau6C@Imo>=xop+XF?HHaJtl-*!z?{MDC#}#wGldsYZV?hVi z$KZggg3Vto>OD=BCQN!!;5+#P#XbM82x0LLDXq8LdrtHwwofz({#-GV4>9$xoTi`P zeV=Afs54iwqxk$OiiA)SPbDgyrX4lw;qNyb(xdi{#Mc+wmA@UPZ6}2fY0IeE7mK zP6^%;E!)`wo<~J@;D!x#7lhGA))W$e>)y~9l|p@q_;_Q^a|!UHX`x)|;v`CJPt^bA zbQ#d^P}F~Mvc;i3xG&H<1U@?5jj@{}NBKANb(?ZO`2bFx#?_Kkomdf7;yo*geS%&y zFKhpgM`%SUyO2|n@MoM$7CzrgF`u(b!C1Pu0GM(FNak6{aR>oJBM+7f8@=^7_eiJ- zYZlUCbp9`0`(yN4At*O|yi^@)!>Ei39x9#Ikd;JhP|Vumgb4pI{})Lv(~1`Jqp@|j z?T()6JO46vau~>H|6n$VA>fK+9Ah|9n4X-*SwMoCX>^QioI~h}2g69%ycasuNncX` zeL$b~U$!FL{~on~Rfru^jC`UT*ouWG9r8pyRf>GC+I-U~gbyFIzyvQu%Qmz;8m*R< z_fad#|0gxzC1t>{y8R}?MwQ7OE9sBnGI{Wr7kv@q)e6bckoHJVYzHBx5*C?%dZCmzvK^?n$nr~yV={^-1M)7cW&%Vm-b8*8{T?geFR}# zr=#B8Y*UD$7|+DTe=*b=ySJ}^r_#AjIRnoDtvSvM05JWNRqNKqDiwT~GfhQ0K z4>P@z=+2FvA(^#uqm2u}LNm0wM$|&W-!XTGbbk|8{l}~2(E0iK>#o*Qw2Y0TQMQxp z5jjsM3KJRVd9tqGK2h~M zHuB@S`<>nM*|ZBO2Jfhx-X(7RV`K@wNWhmbKqC~y0O#Pi)=*E6VtqI*^6F%Ru*RFp z6ay>Of3XW+?Bin&bZm7qJ!ROe>#hjPBwLPsZG>~|3>Z#y+ZfUyLsPzoTX%ANV6oq+ z03ZcyZO{!!|Gf?(v=ljIWh@?4*Uk-Ru3A!{B}}~g#0Z;-=q1YLg^a(qD?w9j-P>=? zp!Xu+N}mgUL-;jw+cE(!7<>2q{+`$bV-i^sS!`%SX=7iw+5~_SP<{dO`lp`5r!n;K zk$=F=*~t_`LN4oSKTP6GCAl{gGc#~DnNR%Ce>f;*mg0cH;1zcWlHKp5L1k=m(!#Z2 z?d+p=zwDzv5|m5fYN{*MBLvIRmZ(=vhh}kb(~D-fuo6&xQrbivKEQN_D?b4z z6*Wi6fX3hZd1#29rfmK(h4B4A|IT5zH?@|5j$mdp+q#;{X*}Ks4vnr-Iu94Q9y(D0 z>^wS2Gyq_kt>R_ign#?AnK#M~RFM;)D24Bt;2*0IQyF&vYS9D_n#icA)=G`}2u~H- zKnUcJ&F^eAV_D>0i6uG4{`97^qc4L{!f=Qn(AwrFrndERN}jQQc99XpO}F_U#&U_I z41>*`eae0|+FfQo>MX@Qr^H-Cq*YF0i7Y)pOIy7&cLf?iGeu>Gn8ItU@t{?gbRb@* z7ytdm@)KML9+`>8jq)6)@Jsh@ECnG??Ty-_HDO{L$kk&MNq(}c_<29)?x+QHuv;I*ca7CTip98Icn9d55w zi|t*$5IY`SCQikAD=hD=i3`eVx=U`#Ahs24VQ{z~S~{FoxeKb7$WcQG2+Ddl0}E1AWq9Z+*k_3#Bi++Vj^-930AiF(ozL) zZ)_dMA{AdtF1o9W9*?s$SFr^g^$GShRVTb&T`w~U?kr9|)D&B6eze*v#r0Y8#cu79 zM)PffU;2WN2?=|?CZxjV{0@axosQG*!c%>a+Z8*9wbL0r69>%}c=Yu|WJ>{MqgPGa z(VdX=N}}(8Bu$bKTz#dwt;PY6bT0oLB17u$m?YoBYEw@ReeTo;H ze5{>HW>tWMZXgR2zh5SE5*GbR6@MRFb^%Fj?6R$cv!$=|!+x%vBRJw9hfkK)R3LC7 zSW7&dZ@L4t?39FaN{)(6Zk2Ryp>n)9npR$0zb=Yxb4RA^%e^vt!JAb_my7qtft|r( z8BNx#dCS+l#=)G--SG4cx~^fUfZ{hW6O(d_qL2K}!6Rj>_dkTYey1SH&8D<2+w@J9 zsv^|l{vpgbG!$r(fm;)g-+C-Z3(r(Ejc@x0zDS)7@l-UC>VB*t|I)(tzL=Dh5(`hg zsxRkr*#~>>nlB>eBgoOq|8M~qz9Zeg+L5G?6qqAU4J8~X>&y)g=X0u+KrMweVDye> zSM~F@Dh9#l2N2J@i?p|Ox6_ce{$D%NO`v2kVnW$N=GBe2-osZC;e6SXfeNtJ!(A!l zQ5XBse*Ie~_BtZE>4Hr*)6r{0BOq~}XTsT-WxONquxX$fipU$>jftJi``h_q2%{OA z8MJ(-;cfic{p5HES0xcAcQfA!h#%4ME%wE8!LvjwZ<6QGn9jy6N@b^)H5hrHjfcou zB=;dC+U)pB4mtwsLo@-WlNelhFF#H!B2SAh{^YvpX$DZa3;>*m(y(T~HAmxVdzk99 z$8~f>M7PpLM$|kbWx9*BI7OUp_>z(H!*rkfH^{Rt8_h+%fLW3|hk{df7FDvkZ8{8lK`VW2p6-u2GZSq0rD;GztNq(Ll&)JoC z1Fx(>Y=&mZ7TIne1ubHeo&>yulv0u@$?hMj?H0bT#>g}NTC8`b=Q4#PQ>44{?da+>zNZ)MhKT&0ZcYH7KLBrK0V zB4-lkY;$SO!;W&Ohxd~3Tdpe3-?Sqxofh5;sR%PY=?m_uaErspOV8e&^#vwUNfIqm zMDKnB&tIqga9XWX@S4(b|EGZIVO*jW4dAHFxxGe2yNjBI02B=j=cLgmUNiH7Xq(uD z)p`k{UM1 z8&ViiG;76mLUEjNO<9cnIPmO-2{BD@veSlCs>@<+y~DnU1W$f4ue<4|J{mt}y+oX7 zBx?xtFGC%l0o`-8%bf^MOz0C$2lsX2l4pPsGv^+u-h&aB*SI#QOgeYkxG?K)p$eu# z!&8eg`(55Z;OJWhpiePgO-`PZLZ1i*yhxN0nSF9i`~JhVS98Zli?$wX?S_azZC^{t zMUlaG)U`ThsRan|=Ljb#6#5B1Ien<`O%@l#STUNM3n4}Yh^?P5bXP(TmV#F7CAry| zL3P`KSAs`D)Vw{HIxa?<(SN>6!d^@;_=htHCr0_JIO5ud3N%%j#X!P%@|=?9S>nWs2@Ae4 z95u|J&bh#VJF~r=gx>J1Be-SSY?{MWy#&dx)$ZICyqz}N3hi}`R{lz5{XlK2V419a zySujejNn`|6XT(wQrxn#<@(2SP8Ys0mM8JV1G-Q(l@W1ZEk*3sCr-=ju6xG}Fd9#a zJC)zvLD=E%&ljUfWLWO07vVBzeo{BqwlbMwY5&2nDNQi(x7hYQg!Hwcp!=V2m`pYsT z!s)$xO7%OJ<*fvrl{40zXt)qSMUCqVe6nSL=yj8-EneZ;q!oI|14y|*;FgN4H_^R* z)yzl9`Q220yKbxbh4524XZ|NI_teVnht)+IE{@85d};gp4U1Nm z&mMDooF>IY!>fjgQ*)CRwzEMmjf0s|uN@f!Gi`7|&&tA+<;?(@y;V6y=(TsJe#o6G zQR(50HEYBQFK_2ObDrn$*+tu@FZlDWTaRa)4KV%g#}RrJ?>T={87@PG0qRG@sdYiA z+G{JWFk!-h+AR?nQ$R_Dx1t&5sai`%p zM#!sqnt|*yE_huFTc@t8g9;NgI%!Iv~5z zBZ}*#;>+yCzFefMhT7ntj_Ew}*`v%Uu*GibQSD%LON9sTWSi zd0$TmO-~ng>30}-<-=QB(!C!+AV{T=C`vc4-A2l~ZD^&`2G$M(yD+V2svm$yG_aqG zOSV-|)kLLlKg~AseUN1L2d;tWNws7jzT7GOmCt}g2_x@ugvD>mSPHU8OM;AuM;lK& z!&9?@s-IH*;a?H@K-0#F;ghp>hQdW{+{sj-W=ZN7evh4UP zdbAwu*U)O7c#ts{ItyG;>!e#_eA@#Rue;j;< zfc`=QBhMiXY39(dFx24jP(;6~dPm(DX8K8x@2B|0sFN6}Hm@Si*(mmeMkp&+eAbwv z<;tcLUCs|O^gkrUl7Z2WozpFE<2QN_j_7(*gnq)2wV zEG`kSt;V#*jND5YZPXpLUB9`!xN$V=N*2zIj~fVAw_i8M3-6&Sj-Yp^II&sogBetG zu~sQD^pi&C*Ej)JQ#35CX`?4>XrhBD1R4K1fU6Y^rILyPTW2We_sizH?K>2lcXg%NMjMf-1nj@+8|CQh|ffUhzN0nsA9KzWFbVV2@varp-B=t(?c8^Ls^e3!L~N{VK~m+z*678_gqgiQ?231m!8I1Ak;nWPx;OTu-kRFZ7j zD{)g;rF`kOgy1XM3AU`)zu(CyTE3)-$^@>#Z8uvnPl z&YyeZ2n`Qwqn#zt)uSf*6;YNiJvU|W&dpeRZ_s%<&s)|x(_*c`H}-t}XNQbqd%T3v zdXZ;v12bUZ-D}=BGICnVx7qQ>e3rXbgDFlN${S=1BK=1H9qIxD+$wYC@krWEk#-pt zJtb6oif)=h4_l6E#@_=IUI$@zX~_B!Gl4(``nD)}2O8`Y&!TSoyK}AQuWlglMeGd4 zl*n!DKlifQ)o>oyXieWfnUUKCc?lSl>2`n$;-o z;Rt3at>>xRcwk3iV7e!AUQ?I@nf5Jit`0Kz{JtMgD*0DvFYcDk17I=g?iB5kpf8`V zcpI6~&DX_uxVyT$7$mtKT82IDsai%jJs+b0zTHSt7Wk7Y<4B`t>>Or^IuGe#pu!=X z4^K;4xekgH3IItJuX2hPR9+}+(ma3{D+a1ZY8{x$c$_nvd^^Nq28Fb4f&b@y7;HD}G5Wp4Ft0dQYgO(B)E zXKHX>-HJix%a+pnm6(XtMcG44OZ|4PD2b{M&o2vGooZ;H(}t9aysKxfbFbuY-$((V zwwVkflsJvzun?p<`{|~098FG6A{G3`Dtg@zf6Kb#%4u6e_TGHZ$>P`$FM{wDw|c>? z!^EM}y@jOrt<6u*+rfLz{E7xUDZW&QB}aaR_R`hVbWz(A-*v~DHdTewY8n!|q{*K3 ztk-PYxpO=r7fpgN(HQ(EFKhCrhm-Td^78Zg^w?ORHr^A1(Q`SjAVyO&2#tcFJXKla z)_?h1WCWOE^Avf->u$IbIHv8J8yCR;iG-DPs&Z~5jk^+PXcemrCgj5afryxyt28wc z_;A^+ghl-ATd2a>trqL84qdX#%MrV~y9+BSj-ArlrZ`6c~8FwZhINH|9;M2{y;8< zq0*8wIU{G1_n@YycLpV4VY67wGKfV5^HATgu(5Gv=c#RwU7Z*qP=OLtg(-AKM@DMD zQxp{D*4DBOf|s`~oB2O}d{vWj}j50M&&7y_k(FTeyI zRU9r1-*04Ni@!s9GrZ78TW;9)_O3nrqJdHD=B-ZQKmh0MTb}x7?c4905B{()R8N26 zndoV^L{}z!#|0~8PM+=k5A%3Zf8<-+|2;Z9n?Ma!zj@5U+ z=`YbaU>dyxVZx8klwcl01SBPKpB@*zZMTmJI=5)u`J#eCLb_fElN9TE5aakro7#gJ zs-5URib8iuul(PO_A$f_^;4xPDJ$PXjIwn*L2Q%;1Gum>U{iaLW$IA4xS5%mU8BWY z8jxyg7tp6!8c|*0CI@+MN5f|YNS<(FQ?Zn*D@-UtG(BlqQ?lj6GroRk zo^ioG(g43GEeIC*Lj+%)h_{Ie*6~|eJg1s2U)kF=y&9rc8PHNtXx>i!=a4v}*tc>v zS$|^E(~p9DNn+@K^;+#uM*6)|Y`j7rW-;tT7Lx}M96*|f1M8HhD&)v(8AlR5mt(MF z5`Fmb2AwoH6_SmW6P*fy4zCEJ=SC>zCkf_|1yhGDdnv%LFQ#TUHy=-(kG+u84llFV zD4`Fx5(YFw6IR2U2!9pJvb%{&NJ@89cstPsYyzfOyZ@|s0N5tSk1Cr3yPG7xwQ@Ns zrt%d>HwmQy2+jNrXHpu|Hdzoua&QMmuOkxzH6TA^+OoVz;BKunX{ehcLF>k%%u(AY z+2QD>WjaGw?z&TkWS^1SaE8rH*lQ6o?H0*DC<1LErv5#iK&{yzjdjqA?1$sN4A0pO ziQ>aO^C;EnY30o?X;uA;LcMf*u7AjZeB~$weg%$8$)NCoEU|_+J|2eo1t_9eGvzV- ztXNu80|z^%rGC$UJjCd|6PKa3$8!cZb<0z&CAJE)iZU}o4=zy6+eqavaAJ3p=kIb( zRoFE?3yd;;36XNs*J57wChAa29XdU7o6Py8Scsv1zFS4HDjPiWcb8rL=qt&0l zHAz&B3}1Rx=kY=nxszy_ZbQVg>$@3mrxs?NfORsqMCeB;&f|$-gowz9eh%$OjQP=h zPmNL88XY6h*+(7TAIl9Ky$y4AzJhzCi&XrjZYnzPQ(*g z5Js}XptWZvIYJDY18aUHGRe}w8$pVO)9JitgUXdbm0vmdI@n!d!XTdf82d>5%`+1}N=ImxO&$W~-DxPuwx#Q*0P4gP>QGARjgA%sMpvZbG+ z<9E4ZZwh_x2bSj$2|Ub2;5;G$zF4-NQ>tr;piqu-MQ)wzIHq26(AOIqvqAlBkTn*{ zrlpi}PW|R&__adNeg*WO<<{9zU|JF{A)JMt(IH9GOgGmbuSudmFCvqA@|ezV61p=- zG!Hx5NNcwV%cu3f@VDqV?qO2xdp!Pkg#v}hLE#~<2*cYenwvI>!dT7c2jpG$~_SnsNM zH0NV5j<061cn(kb3W!MK{#IOzLMA-4Q7-UH11goeV7=jrQFx1aEk9iE6K+{=dHE=g zi%8r0gO3He-<=ZAZN~q0NP`4`dc0(C=YddQPIe=$$CwIHmj`@c23vSZSS4i&zjC-8%zD<$tc6~1V8Hf&sQu! zg^Hye8;rIE>24g}q@phQY#-X66XQDK9dVP7dP~vb}#r?)yY3_ z{0)R9{BtHt*eEFBwgw?DXi!Fz!uC^gWgf-u(h3j*2JcWk%C*Ff=sgCoy2Y?`?c%%~QC}jeUc|KZWzF z+8ithA(?dlP&E<#cq?b9aV7Cq$Ho4Glj>y3(kiq&-`Ue2_}HV;fTnNUBmV0{Xa#Ei zYe?l{HHa+0F@U!0-Z=I_DJ(7Sra!&|4EC|AptnDT64-6y)Qq|lg&QQg}!$X?}t(}L5Z!6{a%nLr2q@-yp)3r4;=u1Mc*V#MZ4wes50IGdm`xwN7wpYZ3nD z6kT&$GZqkgzbd~OLE1j<^Th>|2K%g_n6Qyd3|6#61U{dV;NT?lJcZ#fzF(<7`!s6* z&;8>A|BBU-wxxU!gozoLolOEbW^-hJqNRl;T=qi|$p_HLdtA2nySuwf^9m<(^|qum zA3ru}uLn;~DnanMr%yF3EolKK+hKR#0IL~lvk&-OpoWHqC1Ju4Jv?i`+IDIF6b3eb zJV%U_le1yL@vaGQxQ$InaBk{B+fbB~(;c3>e}f>kP-iWsuAX$A$LA_Dtqn^zGU0X6 z|7wqH=Ov^xl23g{jBmdu^zzg6IunbHJ-Tv;i;D{3e_jntWc2($`5NFyC+h<(s;{3d z8x_2!AR;2d9u-WgP~AT|3Lp`emc~XLX=$*B$&h_5BU6xIOblakQETh6yy4p8+}s=% zlVP7uN9Ym~X_CiVk&6UN1B0}j79e*ip%o!OkOobsLIkzEgQT3QvAYGhQA0g;%@|lI9HEZG=5->7Hm|VZ#^^kPDEO z^Dyj9xNF}(4z2*qUBD}m(eK}3GNMF4j7~?3yK2rsYwr|iQXwRY%F1!0L1aJzN7CO< zmH2~J1Vm{dj7l2X46CVWn9Z5DEQdpy8&$)OuTE59xDg zy`2=yc=GR){5{?Vs5uUeXY(xDB$9>gsB> zUGq;cN0uNDszL{8!hJU+SVk}kFUXRLMLyP+DMY`OBnu06GZbZFhRW*H9E{t08CSf` zi;?r2N<1n`3dRE3jdhmnf|cwogUAs-J#Ej@HLtHDZU`@+O}3|MJ)No=LYn>m$s_Ra zph9>+7Kx}^h7cqDG*Se9nI7#eDhy6+=U{_jA|TD}QevR|{C!*sVt6(c-|^0$qF|Ce zMDqm|H7&>+X3JGq1d)MF-7aDv9&h?!E$}(*Fb+P$(R_&>%2Y)COo( zX#9Ikl7|QxV{(5{5PMqYAEEI8{g{}_G$8c?E-1-ld(4yDAY-yk^*m$|^9?aLD*+%= zeax8fc!0rCi5wT_r5H;LFC`d7WBL3~augU`aO@h9v+XMaeB*S>5}FT%pi`ui7fFHg z?V0jEPfI=-|}WmGoURXP?2#L->$SH*rqi@~t4W+xroeJLH?FqEpX2fz-8p!}d}7+HvL zX+@Vl02SP1fbI3NMpzuMsWrrC&80Ov6RW!mM+NllZg7WiTf{fK$opZQbu)5t2M(5E zSu6k$H=46TSApM3`!-me-2uS6wstj+{VC9$&}CpkQbC_*gyYgg>w5SdLo>suz0N<0 zBqC@}As-a3mrS3#kTcIho=l%5MiHL~0Z%Hyo@PzDf;yVPZ$QE%JG0|ocwj`8mKuQ< zSQKy@t!B(Xt0@aWZ(zZ8n@~{NhOQxE--g_r9qsF;D>fTs*@Z}GvFCHmKTyA6x`Pb4 zE}0U*G1tz&;RA9d-?o6{Ur@*<%#KM<*QW(Ho!?}0avc(;m>7u>3sPNTQIc9Al{xQ_ zGTE2@OXU6g#gCyugXFOAfsjE1aA4IoOrkFv5reC7S!n!Qk)e!eRIp{*?`0SMZV=cK zd)V;ZCmB!XuOq5tpq!{gyKO_zZ>lkX<+AkxId~kh^g2n=UZ7uYvglZk&W-DUYN)KD zh?8u}=I6C2vitL>@|Qb4_&Nb;qiz&yH!sZm!jf969ljS}O0!l*E?E`>B&WbYl*FbZ zO>oafFsc9Fv;i;iJd+S$*t3Jylpd1x11j)Wb`SvDRyi3#w9$Jr$zOpe5+I}0F6

sxL-}+hct{&m+ z^5I-1*4iAdo~bfhM{Vk9%TI?+t(26-GZwr3mPVGz(txkJ9Xoh0CFB<=kyPy4uTlOizQtUa zc1$sm$#|H*@dO>ufLG$UDP*z?F}hnYpx0;`nwtm@3+s`f9O_XW;qT|mtN1Sn{`Vg( zL;{SfL8c;wrB;HrrmPxL+Dgu1#`JCqF}lQ&{A952w!(Ucgb@6`eo5|y0-6XMUj1+| z*F{xpaC_}_Q+3Zf(km>cx1|>&HLzF&keyop3lxA4O?Z2d-+Mf6cNV^d>UVixYfP+m z)A?qa(x+&~;Jdno^^S&s@+%|%m{vlY=gSOmLSH5JCy7iKO>DZ>J+=t@Pfxr{lLmK3 zb!~p@;Yq8U6@TMWUT7cRWi%5rD`x-VC901VV#?Br$J`2oKnqqOK%iy+~qr7l>#%(UIKILPh2%`ZM<^bsMYic>Tu+Jv7#fAoJmF(S%I8M;KMaZa@ zJg@2W9Q=qL!FwCRYHUTHoRJWg0;CIf{q{lu-zffhd>^GVUBp=9J({~wlg@tXM`2ae zB|Q!=uH3KGx1b!;E``{V+A|moFnBwwK3y_GJ;Ll5WbRp=yp27g>a95Tf{A#tM_IJ0 z+eF@H^4#?E=eUR5emlv=q_QOj&nKhlGdM_Bb|X zwY1<%OSdY*t*@_pj>Mf`u+rUjb#)PukaU=;4I1Aa(yJ&h1e}>+^69i_19ncI243B* zJ}mM0AevC2>_|dievq!e_ANL$S*K3|i@SS`onemYO;G-+J@&4l4^#mE&vp$Eu=ISl zhvPPC5N8_o-$Hrz>S5KZp{q#h8?vWK2Z~vj!83{1q8CM>8f2wTPMFDQt zbz}|>WI-XFY{g*0B6J!-uyGh4lu*Ti50}L{d11<+lX`0cK%L zeLpi}cdIP#=CI{iKw=j&gKql1E@Z1VY!O<4lPjC%H^)r z%fu%qhoq+Bl^`AOTMQP`Qs&fmx^jtpK7H~G)|7d?yujno+CR(4jDF>b3 zWS)>!78Mu@t@8o8&dQO`{DVuIQ#ajdNb3_d?oNs8-vc$e+vt1Z64D7S&Szl$aeRp) z;pIy8_EPAf^0Ko#zJTqED}_kje`U3b99BI^VZv7Eg6w>I7@XdTzb*Eyx^npFe1(ZT!W4X?mW4-lfHy!^j4 zT>_i^*JqzK7})puS)AA+??m6+(m;lHa75=h)IB|U0l4!P1Nxdxl-X9wIxFb;a4Pu!$XeG^_YP$4E@DafVG@Liv~dz{;4dcZ7ZL81fP!^(@Z-QGhcF{I@|XTRB*c~p%=Z;F<=;)iU-fQtj8C35ITaUv!f#NI`_cW1@~Z|YpXS$z5J zP&MF)domx!-!o)hE;!#O_~~g%d-mpPf9VLf5DhZcm&Sumxnhs0uNd>=VYT;Uq~e&x z8|+oNJ0x(cA0gyRZGS^$7>@Jp=P2K2Ir-HX+}$7}_SA+YsHk2tZQ{rYA`a9$oF>=tfJ1i?bLdg? z6$ah8XCKKna1E>Bg5^u_`N3`rAR*=9cq`mPv6Wq^PpRR=Vf_m$bz-^{I_(udtIK8E zAn>pfbkDEo`CPGB%|PX?7EoN;=(8=IoTHp$+jP$_=de9kKuCL2$Dadt|KsrJ=SM4N zeRQ+!=(V)alPjAF#0*;r$ei=nhvNd^229xT6N)apE)5^5l$vxFS&E#d8rv6L~w0m+i7{ z2})8c3G&ZR8!;ht%0hVXR$|%gJj2e0dCD-2H+sNIesaw^>wc zzj~5WdQ&smN`bpak~GBiO8B#rb})1DIOqXNo?v0;_3`ok1#gqL=Lae1s3iYh{hh=Z z=m~dM{GTu?rE0$H33N+<^%lqeF#8Imp&)Jh94$H$Ha{3>g@@tw9qM*YR0?q(;?dq} zQRny_IQ%Pdt`6q0`6bBuwgb&VO(*9#hE&~c49Cx8;akQw-^^;&8DsuT>>|s>01;7;&z&{Yj0mq-d=xD~5i@yY03zi@eV%9Xzn-7P%3pM-`UvBG&^)<~|bW1_jQK8W7qEfpvPD^Dn@ zv;XY;JWWQE1djAuYIex1&$tv`42i0$!6f3ufyb1}TFVPB(#_W#M)EEvDHUSCs68UF zxc2BIO?CIj5-*f{@|GrcWalFo1-$He2}n*fE3eQzPso2%f{+q#@NfXQ2HJ}*mq=1u7x{)D{w-m2_&K`v)nG8Gpx;W z2916ZbC<*|X0z%^F39NI+fENymuct%7%DMiTLGW5QUmWv{KcL0-p9 z_X{@8_}Y~zNZRL+Y4&JT>hfS~;c}=?|%w?`RaESFb4)9pQXa)%xh@aW`sQR;!Mtj!V-6FROX_ z>-b1)_be~bVQUV{xu;*{Ly5$8s{fWYtJ+mxr-`XudSF`_A1nym1b8#*IUwa2q*{Q1 za4QPn%jyZd-V1V&Jf2<}DOfZrs3yc?$|$>i&Ag5y|IJT|*a<6JkO@%FnNLa6*qIFX zVggFQYTid@qNkzf+4N2Wx-+ptI@>p;fqD4b9xZNy_su{CqVw#N!s~s@bAM2Evk!jW zJ56&P#niHaDsR}r$?%8bW&cOUq&o)BKx)+&TSOs9ZA?$NXh|1apJCjT~(`5IKJhGcA(Sx!h8Z)P*8?#jcLNR%eg|d+ZgwUV}DZ+Bdc4$ zFwf$bxRQq}x5@bUOe=Tl-=d(Tow7rUvs&L0tQ>Xt2oL5by`7q0Ba9E*wh4^kyyq=x2w zH<3|F=$6VMqz;>6pYht_aG@%lG{l6aFg+BSL=rJ!zi#+}oif_+ieyt~!4S-DOvk zJ$rcS$4oUUv-F;f1&NbaGwSiOMp%^g= z*W-^!t5w@zfsV*ecy4p#TVb|vDtu=!@%&ebC>hBN_nI^M^Ld#1i_`_Wy#rrON)L2q znE6QY4P>}c9E}PZ%B{~htk*LdJ|JJF%m!$R|c55x1C zB8EaWEDJ4veW;g!g%4YKSWRIB8NzEz+%aU;m%(=h;&a;}u4ZHX^J9>%7g7QC9T$DS zV0Bj&20i8N?ZO?B&=S~Hil=q5u^#^iBO?j$3w@k8399*Cpe)PhFPr{cw={IzroeGp z^DSBXU*&O7$c+tZ@&Sm2@|D#6TD^9T?ZN#471e#iwJL&QZTp&M*r@tUzbce%KHOdM zN7C9vtF4bfYAV<2bjS^QXE~k&h?P4c0sf6OcqXmIPR4UVK!U&DJ+WA9z{GNuQ zsp;rQlJ@qpS}X2;y7yb#qTL`Czt5{tRVe6a#tOYs#HOU5M3VFeGxj;-U25!aX+C*g z=#_*ncIJ~<_YfC;WaUPwt0gkt6W-_9r>yKzd@2<}iA#Lt@>Y6rj1 z!n;7CTT;wiC=Gk`fR9bTni@G>52kpDg3AoD*$H?307ZN0&Q>_r;D!EW1sswkhG!I( z5SbRNCq`hk$EKX(z8A&n+U&Uel_V)y`a=0AOHs&(P=NA&x&D$;h4{Ffj&ZMVYC0%3tFtk^)+x)}+^T(6QMTj9T%e|Z zwJSX=q4l`$uJhMH758~DVG$A<;t^Dck&b;FHOx~(*?yHE5I(xD^4ZFoD zy!0wXbnrUkrghH@9dn)c5u6CD{qooEsLa?U)XzoB)l>7E;l0fXBzWwuL6BV?Nb>c~ zH}WnqQerwBW!AF7a(iN?wb0wyr^9j@*y$88dJE^Nc%5o!C)FB#MceD38Ujb7c`MqY zqw%t3cHo-cp0ds50{06KW7o=th=}&u=~_h6dVf7}jL?-qL7??IUrzJ%RSG@&1uX^P zvoZfY_OjbPZN#;|Q9@WrMJox;=3lZvw4shsxzQ!YUEfJuCc=k^RiFS@wGZxftLo(h75Fv50II-w<(&N+|k zFG^iA?wU@egq$w(UhB=w^N!0UH{lwV0UVP z?5b+{b(S~2Z$K;1A6y9i#i-^^U1h>jTxTf~h>}gE{~?L5f1 zjY_*_vN1k3CbXl|IHzh!nD$ou@ytQ6YyfqH$M|87h?5bcDjz+&vG#3uUcP`s$zX1P zqpbhM$AV3%HdHL-#NhR9NcV-r?ITw>*MHSPQ`;~E%p$nQjr(*+NLJ^rXZ@7x^X3&I zUaejxuI+zM`urjVa*Ha^^b~rMrD)Uo64eYN@FO6rIYV^TBJk@YI*U`5MfP9Rh159* z_|-#)Y;zdwo_MP3SV4&mEoIB> z3>8xMGZ+Y$9-jv8uxpG0%Tca>qQ|ef+0#{o*wVW`C%p`?dP!mD;ZKFD=<@)-?%%i{ z2iwFK7L50wTGTjLKG++FcTw&S%4w*4i=V!F!k25gr-jN_AYd$mXU;!JL)foF9IWI6 zTI-5Nql!CQN!r&|kdm}HEesgja>BU~RJCkj6S*k=E~$q{hSnwtbK7FXb;vf7WGt3y z$wBEQwER8JG-evYYz!eZKtDH<&JW)9e!IYTqbu8k!+jL3qtakYs=|K?3U5ALNVNOq z+DE_+6_p?4a#Ssmd2`+$BkLSW_NA|nsk?zvlQoyE`AkH(<6r@YIB1*zL6&NEjen^C>;MoKmYX{uRWg6?Kcgu!wdyIzMnSnw_=3Z9*0 z4tZTnQwVMP;a;8PVBQkxmxKf7(?gMe4w{wf$7HkOvYmWJN;-kz086#=4to>xTft8Y zO{eX=CC}bhBd>o9oLM}pULg{vydoBu#Wnm0|5I25+15{^TBU4ydVGF*-rI6tm0z=@ z=CP}+MkSR)03;u9z}o@uD*!*|e!StmBFocD)(nSAL~K*u#G88EyzQEC>Td8yYrFBd zpdQ*^WK+~|px^ueWZtB1lY1qQDdfXm+nXlfpoB~!_S2$6KHS)i#E#XuP|#$148dRtAF4Kzt5@cX8hF zIp{nUe{e!Si;g)+Q&x>9J#At=d4H0yPf0BBod`W!!TAW)*M#~v;sLs|oWV?(wJtoW zY8grv_1jxh{)?~s)pA>o7M)!612P(pQ;t;P{JymGtsy`Cw35Jub4R@QJ9;knFv%AU zYt`?I_LYLyHQ5c$--V9d?_r-rypK83imY@dM5Mh(=;y9YW^?UD{d%C>$LZF^mYM`M zdhn`;b0c}~vfXWaGxs*o=Hz@6J_r(7&5LPk+Az0}QSogp8o^>d?t9!m5^-B*7s!_n zY`3P4H0I#;Iz>v8Yvrj^xT%01^V zW~ykLhQLK5AAJ|P^%%w^{lnjV$;1cak6#Ua55lr=QQhPX=BQco~7YR`Ok{wwil&C6Kx40M9 z=c9g(=~=_Ho_cnF)YYfp(`be3RkRyp?SS+3x?pBmXVaOKM#NK0Nc)JP<@~niLqtR2 z%s_XVlusL!j8O;kc2v7wB?tfIq~gs^PQ=is3;GS9U0@v;5|`cQKydsnkOB&o=-)rz zbBWBK&G1RV_qfto#l@h8qTsL*^_Iet5)lcVcvrO+Z?e#AZ@%tc67g$m$Nj;Y|$ z41+I>9E$auuxR0HU~?*q`*owamZ=`%mvy^??U9Sj;-8>96MA`dbyCE5oEw*hENcjV zc1LpxFDttQ=_FSv0Rch$jFOUgW2Q&2e$9Tm7x5wKZe%zFjogm^!mVWN6dJm8qOq8K zvr=l&QCV48B|%WqZ6~!%u!5SVxMK9{Wox);jf!?gIJGEp$>0!cqsk*wKh1@PXLW-3 zXpw-54!RSec;48gj@1|3kJOCdS>o-YMC|OCFSC_TpYD$ryo>hfe*>N)t66~r zKUvqA-$uUr>sS2_`YqVQe{5E*Xq*LOsX&UMY1S1MHl%a+$K+QH3+nLd(BO=936R8I zYW`@NM4|44_MX=5{k^6RFpb%kv~^P4vpgUREWXdG-Q*+!Cy`-+Lvw={H7O}32q>G7 z-7fd0E||`_`1uKY!||)UDNJX@MJOIgczAdoa?@VbsJX}IdT^m8SK|Q{2S{Ff-u}c8 zA~0y9oRN!bK1l1sc22Kbn4Q^&ED6pMtisVdxzAGD#pYileMq4}$#_w#w`}B*2?@pu zn|wEazKOOGQA#EH0At>2ceOaM{<=xvhFJmy=AV{kPZs&dJ7oiQ#$IC8PbHF1^Q9Sg?+j9AKt&Oc3Y@Vva=iE!>=;krs>&j*`2M-(OZ&JQ!Iytg&nhX zY*7)^op`#LZ-AFGu`9n{qEl@GhTk?nUM;*dc8t=ieCmtff8uz9L}0<+`}#iz^8X#n z&~H8=tb4JxN|X}#TuDhuV{Q`CLKR^lA!cs#&C^$ImbR4`IeeTJ&;$L)?GQ&A%2=*w zr4v)CMic&Inf!Iu2cC%^-evu{<~7B`xrdX)xg)eCWBXi=dxmE6>Ma@Z@lGTD1n842 zymJ0LC7k>BG|5GBh32Z`5*gTJx}-KgLFVp(qhw=mU}RD`eA}KbE7U6W{dvEZ14Fr8 zfML-;bbtA=B|-)MzbNvbaAG0}O#>>@&C&N)E)12xmk1BBElH6`V^hgO#B~(pEOyL= zR#waszn51HpH{IWlPjvvHZVg_pO9mZ3DKmfSM%2?C~{MoDej9+O|@`LGOb9(LnoMF zEiT#73gy{rRbY>a>oZI#WSVT)ZIqcBnNA-a5%Cd>v0plNs?kthS#eqaK*}_CR#1cd zrxS@EvS8Izrac@NL;Roh{d>6_>@Z3;l?~rY-ISaoreFO;K-ZqMG2o}0kY57#^BDMz{)-^^=zWoZEr6PgN!q#Y= zeUmy4bVUcLkDo>SDTaUlz?o42_shFwBIt)}$;iXwu0X+Zna_foS z7BXb*n$tY6d_Yf9;;H<~&Eg5(|Hx@aR`e&^3Yh&wL?#oR9B(KL6vPk=fXy$_0k!EUir{ z=L*IkP^)A$b*uX$;L|j7pn9-a4$Fs;Ig;(x(Ppkf}=0pdq(h&^yjD8|GGVfyU zgzsUR7N_OIYfP#Bu5^%+)~iBzTw6vVExDrQEuj18=&U|mrUz~eP@MX0HXGjyY`gy0 z5yv(_3v+$E67j;Zot~avH5B?Q?=zWtQoQ+Flx_EyXz%Vuw>r8c>@hbl?nbdav(m%d z8sONs7ZHFZ6l>}(GRg6aAcfjgvZ8oo(p>r+o+?K(ulAOYCbuxtu6Sup31qJXmVm{6 zxioVNc<+uRyB=k!0((Zk&GEmwrk2ayHPoxR5mrCNoQ=B{noN(q%|D;H&#lQ*GWn^| z+mHx%UvKkP&ZN=icnOx>z9z?PU~wSsKX<9v6g_R}buOEP$I?IMuQdQ?x9&wN-=5-^^m6e_B3tT2$Dd(E7 zn&03$_7V7gwflM{QrECQq1?4MoE2w=o&)b!<$nzN{_iZ_c@9sdxcU@d^lvGn&ZCBxtJ1!ZPXrg!l=B}gT5 zQR2PT#TB{G!CcXJQxXbM{lKn}cC}aP(wulLTcepeq6|w@a=`*HvdqzO^?4*QA^J3> zHAaQ)hLXdIs&RZ4f@ve_Gv$a3+JOk)-=ms*@ex#1B=K0Hfw@TW1E#_zLjS3SVv#-^J}#?;_$@^w>N--d4EGw}7?ZVqU)CyaqpxtVh`Z**i%# zgUN{2MKb3H{3fx`Au^ei$@g6?=TK{H>b`!l>nE{D_e;I9?U}CUW2RlY^{#H2*_rUY zt&fOJEAE8YaO-l4%5y=M&n3pIzk9v(Mm7l~x&q$;|!dRE+Ep|ZDC$YJKl8o?41X;#)!sO*jM$*zgJ&?9FlsuGXuqf_!mR(p1Q99MB)Q8n^-9&GqResK{KkdP_ z9p$|y;^Jyt8csqZ`CG&{!S)TwMm0R<&o#mW70h^_MBs3~Cp~JwZ`C92<8eXeln|4% zj~7&QX?uc)?0wIa(H`I5D>^wjj{;@blj_u1bQf437VRnu1$5Apfp{r&!h?t(i8H2G zw+^Rv)}Ur|LQ(h|@RvX1%O$71b0)O@esSb`8o^6-Ab7fDKT4vssA=HkhSULfDfHEr zy109I18pAVwRPG%S(LNsr%UfB=ZC;BCoZSpU-e+VSI2)4jXqTz2-G*aCK|MzFbE^! zes3>Nlhc-mb!43DSeohO7_+DtovLb#R?7W>>aX<4j_hXVhQ^|>T_1S%9C`Xd%7p^i z)Zp;DX+C^^gpUQNm3R$F7LU8eM4l`k(n`Yj#Z460W6?m^ziS0xZgVM1|~dDbh}$GbPrr4oCDGAn{m498$e9MZvVT@&}b;p*ZsH!lh^&q z0`q)>?5~W=2L=yq4C#iLSD(D5Bru3br?4byDc|l~-i8F$Ribx8Q2LTa_^#a3Ch8eI zU0g(N#subi2a3Jou2FjR^<5SNFZ%BNb*3m^y%TblxCojviuRtvb#W?>VJ~GS+01;J ze#+91+w<}B5DAiYm`)Ml?^~h*!jv%A2T({T3dJFkUA8r_XQh$5xdDfpoRn=MREXV)S z9;H^A)Ai=ILvyRbZz`B}!7wcFP_Y=LSGp+Iv4#x#u?3kZqSV3`b5l9w#QAKBI*Hm& zfsi&Lc8CzB8bdoxJ^V(vSO0g>Jx%XwZnGl0?L|4n{iLtRI~-p-In`@%ToZHh_GbtV z%B>bMfxhTQVct`5!r znUeZmNfMd$F}zP{d$m-N`05xDDmt6INJfG(^5<8$D~}&oMNQ60^MZ_My~SB?=Xw!? z>rwreRb?>8b9HpF9nQ7B_a#8Uyr-Hl!ovE0QuoK5N2Fp@R4KbV&(R_bo}=?-+b$GA z`J5PJ#b3UBJun1O6cFj}_!u==a|w&;O?GgHspXVsSwEi`8xnqLpS>p(D9C#K;olwBoK6elkwxmBTdcGeSZMmt#4(NlktTm|xdAVBXFg-nL zodTymUCaBdNBywGJUTjOXL?%*Ig$O!XAAd3M?(UxSM&{-D6j8viHoup3I>DZcf8kY zNXX&@T#Ia`8I(REBQ2~A%^iqz7sJSuh-#iPbOd5 z53UNBdsOnN0c|Dj07W#aPer0q3CO}CY6VbStay}aJpL4v_5Irh!FSTbtDS+T09C99 zG}2sAo9H#czQfS{ohxx=|Bt-Qqz%S_vO)4li>f@OpTd#)>!1t@b?y(xm2~)!UFeF; zR~}dGuTLT!=$}WT@nzA&?XyIAq3;WjB&E#vUy@qA#sD!9(sz8IAD79nw} zmk3c5Z_~j-acZiL#gVOOAz{@momh2xHfBU^z0v4G#~ku{1e4UV9DlQj=ZKkXKw~w- z61>0e$!xHXN=W>^V@ul2xe;-ZX3Oz&OA}_GO%;x|o&aCJF?-?rDb*-lLtx?LNzTi9EFlC`4&Cw=!*p zfZFewbd2j&b><5H(<#!r$JjYDtU-89^=l07ym~r~~&@~n# z^nOQ}Q)vk2dFR^>@)CCEal`OXO@212SgU$it~9Q^R%Ebeh6f32!W$wB0kAF7aCB>N z-kazY38=-3iUc_wjW$iKy(fGloG0y0auGRU_PQRc3}Awb+VyqYLOz15wf9BSof+v` zk$6Fo9q?_)rlOR_Q#d_A0`$rY$L)U>tQgA4%5GyD9Vx$5v4{-8dKwS0m@C!&4%mJQ zHph7ET&^Rim(a75o|$mqdX`xw5%f>qSE>(9`U-ekL$D@ch3>^TDPNW2fRb;7#%N&T>9j*cLNVs!iaZ@cPCmT-bpl&QDu58MkS%T|dU$_;mV2}m_AP^!xg*KD03KyD*DJQv z{24BT#P-o_qPe&f1Q$;$k}#elbx7*?Xp(z8A1Xwmgr1+qP!UsS-Tg~8fjHBilC-{I zH#xVWzMGKLbPy@rYob2M8%9JVX9iu-X18MBIpkvkml_;D*hrW{6O+!0yv(0eZ}OlW!Xee)q+`I^X3CTz<9Lq{%~|=Y7}-5NRauA z`Rv!w5d*L)TyNjXrqV*u-aYn_M$9&iudO80T)&iSb40^_W^Dfh&D>MUr6vUDI;J4c zHdM`Relq&av6G{L$0t$39<=cP(e)KjRc>q72LuVFq@<-oL6PomHn|B&>25Y1(%m54 zu<1_ekWjk2k?wB(mvip9*K@!Bj=>ndW5Byzd)2enob#EFVclMa(#)RUm6%9iQC>@+ zV_jblnak?y7#YJN#`S~UuwD$GVfh_A6HRjUbwY0MPO}!?lHv6+Mnrc*ZQ7cz9{Ez> z$XM@K9OqOm&i)wZH?Iavy~Xny4n zmz?PIW6-{0Eboop5^v67NN26e62FKCOt=*N#ONNO^xb*chtG@VF$%aNR&QkX=IPFp zk~SsjXaGW&sHfa?O$Q4*KB%*#!mP-TX8I16T256p_~a+PQ=qC_U$Kr{0A}akAlFXO z0wtH_9Uxo~1ClK9`lMQBzU@`^9UXO|>CM;Ml<@K*{0=Ly*THc&x!J6TSS=Hbk0A0j z2&o5HYWCX&9Tc#>OnZGFlm5n`8(UOg za;}yjpydX6PRny)yzm5d?@K~Oe4^m5Yl)tyYUgLswU*2>6bmy$>ZZDW-@CJbk{aI@ z`*ONKH}_=&!=+0hE)P*F^?NF4I+v}uYg-L9y}T)J!Yl2ua`Q493SmX?@bYm2k|r%r za4H5xM*m1AF!|&+{p~%xpNPR0@h>0L2kJZp*1Gq#jqHfuiD)}Y?`gLH<2a;j7Y)LY zRk|b)a22hRuUOee+?%OF@B|2`V?Whk7G`Ze<%bT}vk@^yo7L}QnuC`Tzim^qq+UnU zEnCiA6vkm$IW+|Gu}R^>Udtl&}jpB?LCxR*vheK3Of$B0&6IK-daDQLh`ap#&V!kuX z@iS1`MRT!259%>-smQ&rC?Zl*@MC?A^6}wtCm_FkgG+UOVPR!+6XD)_mhR{JVw|{X zdP7%|zsKV6E2724$u(>ooGcB8qbg)siR72al`^}d3ssPQ$pRQd?3R`m!$DSDtxi@K zkcWpyr(#$*M4G74B;QP?=Uv>bF}6!ssgm<2KdqvOi&sDJ40?YSb+q`}BZKyTL917| zi#;XhtYHavR^OD%zN-gnUTSC5B%a0R_&>n5JU-kq8vKk6(Tmc6{}0^5P6`9*J4OL=f}d%M(U12D!Ec7S!~hRXS%YwYR!sv9V0>#?P&|mYM^5dK*dv^ z*1ba3yMsA7MNLYUl)Df5?VxTGd)+Z*xBF!3$e0@+yL`&&npW?<*cb+znDHvr0?%>h zb;JUUUnar~xYxtPycrS2B`-u5oN-&OPf1n?2)%Xc7U{zCKgSwulBxSvyqWQ0;CC1w zBbps`IOs+ykeYDZn24@r?H!lmtguPO_|buI$yA!Pa%$B%Ym^=p+o2d@#VOGmu0y69 z=AGA?o@i$vkJO;WIm&$|EFwUP>QVWUI)_1d2rmYGwv5f6UgOK&{qdH^H5^9roiwNB zhL`)tI-aBJkJEt$E+xpMV~;mj-mYBuUpP};5V!}5AtP(4#Kr1tsvq)? z?Lqm-R~1xgo)5Gxj=x0?4gLr}WcJI)Q*_?LSbo2ys;nbY-HQMFFid60qSDa|6gKy3 zzW{#4`&x!*PRZyjhx674(-iPCs}@uV0yZ+6g+k~U`!*!c%V(LM%$uIGyPKb+mL|q~ znO5ELjdP!2j^#{HzLbB~(8Sn)&>R>i{*BEq(;SjOA&+jG({jxsnRk#t6XTT&X_M@l zR8-IA(KF21y7*!XXvsH>#8qBZ-4sm-gqkaG4`pb?%fk==7KuR0nPgE7K44!H{GB}; zijOAh&4W>$Vm&XSvW@{T+NcOqG8L?xIR%e^kpA>r0CqMaFOu`30kW_ZXhtsplQtUb zNcn5&xN&O2Q2X(n+qc3?d{IuWhN z@iPgYf_gU7++)8fwj#Om^G!l4?84g?UuY({<$K7Pv0Hfjk+-^q6xp#pxh;>vb#m}J z@TJZDIusG&LEKs1(vo z3#KF4s$o$28NM(2S&?+>dz!^6H(gulBg5C!rMrFx_lMNokg&Op$oN>_=EK(aTN+Ui zIfZAf5kG%^q;R{hnMoXSC|$A3#d;j%B#bB)LZ|@nQoNDAGQB|d{5xTNI>9vdPpI#% zv93?t6vUzJD{*h5+$prCjj|dVM)xSW6Wo>WEc$RI@m6$rAl4N&`iOwU=iia{Ry65#w9m+{UdHAH)aPML{n6S zCQ!*vdp_(naG;_x;6SmFpsQ&2gHwzP88m25JWZubC~O z6e}sP?D!bsPD1aMS4UDvNmNvs^GkbvQT1_`7NDrP~EDTItiZC^zhy#W!P$sL46|vncM&z-@3hGh*Yr4ecLkm3jns zLgXuBq==LpGc*H34hdO#BoAz@Oae&0vKo!0o+swzjsRftgvNW5Z_I-<=Jy)wCrvMZ)1oAWQ}1Pf`GES$q$##d1w2 zr~1SyWK9$yj-nTIOjR~IuIdEXCSV$1fmMfbmJv=s6nLXfmQu1g!F1i9I9km0(tlKy z$|}BqXkaHM`8WJHTbvVr@&#owPKGkG|m#@F{ zka!YS84X$5csBiZ!V=H8eRS)ZyS~AKt3FT^5Uq4h z$@;{!SSTb?5J)a|!Ir)M%_2)M3K>sFN9QY1k#S~62r{2?=4uEu8ep_|^Iq|SGfe5Q zO!(*~w`#)S=&)Vv2a|Xwna2lm3d~%ZmGWJGj?~aSc?zm)8X|iZ@#o6W70IbYH6#w! zB#Xl{iUFvvbN&}-DK+SJ%r%$rHAM!T%*9>Q3nXTrFE~A}Kuy&O!vXwT8B_pU>eP$i z@&9JG$(oJ8`d>V%wJ7XP;1iEJS(X)+jW0p>(Zm7}15^lg5sSgMO{D6@eav&8Q8+K& z;oNp3nov=Q67$0mW1zv-lTiYOODh8-F753S4;79^?J(SCnBi}tsKK~FYO@pc<{u$#B;MET0mSHel^AoJUl z2dGqQ0dzS5!7j3=#thgO-beyk43>j8MO}rq?%UzhJ{)GXfS;%1e1V1aqYp z%B5r$d8@JHVx!v*A!FbVpwr4%!sctNDIxo+n+1BDH!0b`Z!5%8d2jrhvTr1J#4Bwr zX~~HcH_fmCPF!GLW6H2HESI}@!@#w5rK)z~y4rtv!kUm+TlJ~uR4;d%29-U7fb9sY zto6Yqn|rlFuV)ny-zIii;SBEPaIpUt{ekuYr{N5;2&g{CtN@75DqpHcUdLpCevl3z z)9ks#XpI4$Tr8T?3ZPFZb_{L7$rA7+C6oqAz3L)35T;LlwJOvx)tZaSG?h6)10RrCO*H|ip%>>Uxz ztu{uh=^$5)oLYtqR-Q6-g?!|YoEW?2oTZn6#|K&&`@OqQRKxV3Ntya{6is0F= zti}fuUw?o6%TJxv0;zTMUGng>nrQ!_@rhmyH%MEx-m|h=&Kp}{VqyYBc?`9zgE2zQ zKbg~~F;>JJ&YBK)2vLlZ=t6l5BO-!ZQl=kwVK_%WDBrM901UL*MAUlkK#m<`ov!j* zS+Qk4FyjFF8rZt>i+!oEYrq+muj;__BShy($&-dGbsK4l0#dvD+f~TgKjlR}U}u6~ zjB`ldDsO%C(9wo7)h#}#}^e5j% zo+&X`$d$hWl|m_-8Zxw?{5aD+alMsb^szJ~_B-;~Wi6!3h?)%8*Mlq_q~Z%(Rrwz8 z?3w^~zzgj`Q=(KBGn|8_Q4K$sv-G2Lxb)~zNAe z1M%!gS+i{1sTuqeK?AeXHJt0gRPJ_d`;>Au-RxTTDO!y`PbSq=VWYwE(1Y<^jM;Bc zN*;NnrqZgtndJa{{KRB?-vm7Mq0wYgcG^^27hOEodBbemmJnWEDhs)&8R zlsg4-b|tWau_@2&e(R=yI~fKB{_q3!PrO=W7-Ue1_j$)Y*bfTS%YJs)a$D+-A#k$a z56JJOj}ObsKioPK|6}|9!h|{O$(9F&u3m4Z1Ik{ej7Lj{jV;0%n48bQAF{K-7)eq( z9B?i4PZYIPJ00t~nk}f=0T{oiesEx5`9Cp9;P08d?r$&k05#w+{U)2mtS{ilY}@E) z^x(BT3*a<+z^a*%Kl;4T0J$2l^6*(-*MBAy;whWVio*AJJ;VsuCd2;DbOY1jD(#Dh zfa}?oJXgfy{n8`L-0Ym}U%+%q>>mJo)O-wzFIno~zFv=E_qeNf?(6G&@gyH7_PEk1 zzj9Pma|8VrM3?*FP>H#X=cTW&uN_`K6`4z7G21J1{!^zk2&f-osur)G?=Ch#-ag!I z@~r_-IE#^{rshbVED86`R`zQy2XmjV+!VjD@jx3@EA^4U=BD^hAXReOs@aqD&6*RK z^eG2l*|d%Tqv<2yma5kkjO|cV77>9)?7WVl;V@$mMsSM#k5I+V;RU#itr`9t9OMo1 zoirIm@Ok1;QUJ$bD(aS76B84s6+hgkB?NdN;J;#EV0h9ARD5oSrKal4{auz3`Gv*n zeajVWqvK(%hvMMq#5y|4(o?fuJvD1xRGt1b)p4=C(g4uttC@Pb_9rkx-_+m3q5aN% z1J5tLzJC{144@?^f7agK4!9e82LvG4I~vmSWoAN>s#CL zj_V1;25xH^Mnh@;#oz(0$fwR=0Ri3}Ho(bjtsQ~nyON3~yr-wd79ra-`}-)Jp%~Q2QPB5@M+pD?eHhLAwb^i{FB=S2FJ?;ES`gT)7!BY&&GpDHRTu!cpzo*= zg3F**-HS#0?)`hVB9f2#h{UdYjHh9tF)PcqzoU)f8K2;Q0(>8wPxO5O=R+Er-jk&5 zu{;6{Nw(RF3H}=M-k7K8WQ~EDF>H^qG!K;;OF01o1^i+2C9!#Bv!{$=O+bow*yWym~_lB^56ux)hMyE3(#JNO+;P2k%za?LueSm()OO=8Zi%~e9 zy)2~!q`JI6QX3SqdZ8HEUWnT+7cYMzA4PNzbW$McO z+VSF3w|xlU_LCl#ya^H!w?_DPNBrxx-``&@wKI|`Pu0?vlM6-r>gH3obPwz7>}U8@ zvr%|i?6j=Agl!P02YmEt;4TE4pxmP8p=!qmH-=vweG6j@@6@WHlHPyy^?wHHuP$6Z zdpa-s^1*gd8eCmfDP3(!k-Xu}Y(Y2w()zx%*_6z(@fV#i1es@7={;^HEl(bf4?sIkW%bm)oab$tw>y#Q!e{Z|FM+*jEBWY<)Sex z@h?uy0jJH2J)Y3k<|EYAi^uXg>9RCUn@09Tf7Bg!fzpte{|Nqn+luQy0P()z!+~Oz z;U{@{zgF&B9EU?tc^w)GQb!Yq;s?kH%`L}2oGEW<%=%P~WeoTg3r z;yFEZT|O<$@$vBl3~uguo1Qx}>w&oV+K#}z=-)&Dznytx0D5Pv;JSLwQQQ4gu~?%S zJ1$P?f$)JO4khtl;=Umlb_^;{ zU;Ff32$F z$){mvIj(wbb=cA=sy|z4k12y$mO8bjN9-+Pl7K<4%LEB}7?&j)sFpyhd4V={|U z>?xt~T92=#c(wiJn*^DPdIQn9PnO-J8+DoI7v^VwNOPe4e?PRIvV z?Xce&KTK@B#`^q!qVV6}fqe<+y3|81>)>NFO2VGGkoYzJJ9*b4n0uGy#m3T3%FY|W z5^TXE?SFK=bqk>N-XYrhhECA<8K?d#{wQlASPcUQ+wSx+~f?g%{C_wQx@_cC*R z1AKR$c|R(<-mhxXhfR;zyf^09$RU#bjTJl%tJE*bI_GC%5~85T3F)zFaN?)WUr0)uUC1n(Z8*s~eC9^XIk5k~1Nh2V8lXh=h&T6@fRMZR z=tQZddPa;&O*Oa4=&^_XuE_!?i_>rBpI3|DmfS2kFo*V+OUEZCbb1bioG$zx$#_YB z@MgC(>sj}dsOa%89sKVZa3R^&+$__dxS^z}6!v&I{+1oxbG&8mXgIunrGd77^&|>- z*IEV^uv2NR=2aFjq9#;cBgx!t>HxkkNR~ z&;f%rIdyec&g5gY3i#ag4(YpD?dAIGU zEoP`(YO=k%8^%#Zwc_P!C44x&HKs98H}}>Hxahb6w#n?a>xh=E_YUI{D5X_FjC$R# zD|2ttsm6 zV?VdeVnm?A$y1UDvxzZK%7uo3fwhzz+qye~B!Za7kz`*cYXmr%;^hcqgqx-nudc8pq!^vi|Dop-r z8}d{|8{qFGyh6glx`5;ONS&2-z&6Co%M0HU>=Dshq zIa*~T%SE2r&h~;!iwha8M);y?E~t}M zvwGi$Gv0W<+*`s-B@>K{f6vE8;6O@Vr3K*VKmK^RH>FwXXy(Kb@!vCY8E}_Nk`Wxe zp>Fxqj4caXG^cCVzAWs&T0RVUyi6wPLue*0;e(+jCS_v61k|O~7#85ybno8zaz=v1 z)tX&c8X&vRu|EM{OW*e^V9zHMVl$M;)Keb}FqC$8fVoVE*E59BgdEs7WM#VfcN+D> zft=cI$M7d7Coc`KYpM;$^5wHLrk?mrc9*k!JQX5Ix{H^@+^>NiV`FIC``E{o-7yJV z)|L+sBhv5pjG$rAQF^s%L?k36s3&%IR#sNV0rkGwbh(abzsu!7^(M2I>&=*`|=+`cZkf(JXu!`QJg~&(IS91P%`2F-|K&kYwkJkj0*EdRKY+$Kfe! zM_}teY=KdcUPrYn5xNi1Qesxf6iXG5kdWy1Ti@Oe0@9OT2Sg*w?7T!qLSkGEZAgB~ zx#>@0c>!e1U9FKk8qW*tyuoL)Bm~!wa!TJ6XGI~i>?kPI#T;Nwm1*;x9WE_N^LM2{ zb|(`V%bHFk$-(s6ErdXNqtNDHDiJbh))5c{+3;@%2tjo2f4%o))J?eX?7~)g-7ewx zQ8OS|k4~##ZTnZeHfvTwu+&cBD9i3m&0aWPX0Wu%s!@s|Y{6)yvt{`D>@Ez!qu@fMkY^^i55 z=cMX-wYc$R3-925p(JRC_43^TKba`%VZc^t)IfLPrDNwtbKsmO+?p5~ba>p{)?ECz z4R-hmK~NLiC9Kvwo9$7pvJ9vfE!=A!Ncf0zIS)y_XF*d-|AGvn!@57q<6{x*2W&7* zeRCI^UDv!2h(DC3Yd@UCPq%vVv0BXP6T;E0=g&zj2))30*M`D2|GCut;M->CRSvDG z+8-Js6cAyP+1c|c?TADsBYYJw(D#fcqZFrdE0VXPU{36Cu(<+tRIpZF$ljhgA-B^o zI~Ndi!uVN$3;D~}AKK~@#tY)2qI7PQdw|539p6z_m;Ex&d$X$2%SBhT2=0?Wj1+t6 z63X20QtEWKR{+4_bh}@73?#eY^~6JO7-vbmMn`SkYHXPn-Hd2QnQZ&vczAe(-Oxvv zK4M+Ni%feS<*q5-=$%T&|Zs8n}ujwKlgsN@zy%tr0) z?Jb=Spi}H_SLHU80Axx2@#AQq8f#u0s7XO=;tAc+O!kMa6re* z-Jx613=RZt$>gwIaG}H;!z(UiBC0pXL{%VHi}6-jN2kEkN}ch^Njy@iR9P0XKOpPxT^c*1l(a9EJ7Yxzi}#=57hieFAAW;M zIfNYeu0~gi5`k5w0^6$5(k0Y>&7N~r$ZE)Wc<|Qi0f*jK*o*n}+P@zZBip_K(EGMV z6gP-ZeSK~{UQ62>Qz2&xCx%)}cjsf+>&YpH;2{g)wuOa-@wcNZ)q~L%Y;2x&3^f__kd zc^DrzRv?v5RaVRS)I|jf8{?hm%Bjg@wkIq()PDgAsltXRZ^qDrjYqM#l8YqM&r=4Q zGvz*;4Q-xEW#rdm?Xn%TPwwCDVdWx^dI|jP6(!SX48gp6R%O{`yZ(J@hDMEJX&Apdv#JWUKdR zZq24|7dq&%2V8eh~bzU#9_2FS7`~s;sf|-=vKZO>)to!>7vAU+$XX^-eE~ zkjSpOuTvw3z1Y+S#z`knqNcC!EwgcV1d(^**iM&$(dRCf-%s_`JWtoaHn)Hrk2@hk zcy-4;+)c?EJe!UyovVei17?zh+0V#?t5|UO!4fX>N081w@bbuT^+!;VVxB_h@TuC8 z#4aHV3Pi|+E{QXP*>C@=x+UY<-3po~Kj73mwvp#nB12JC1BjcRdelplQZZ`@J-SZSR>&!Rj+d95)Mu=Q1GjvQo1&paEW_jJeS<&P!tp^XW4(j4=rG$K!m0D--k%>?_Z&1M8HU@H zew)85)i}APyXLqjRgy{Nm~-rs(#(1g?X@+`EU2oFV%C&#d?^j;>lo2HZ?&uTP>e%d z#7{zQ?qVdH>A#@@t--cE1WOkDDgY{?k!YF7%0lBytV*%IpS_B`fOcubH=m7w3 z*mIT5XwFn-5l{rn#$alQx>fT8dy0AE*mi&*3PiF*wW6awn7KrDmw_cLjDT%5{qZi( z#BP9PD&efv{q|x7u-D+KHJ|Ywj|G#zE{Ks3G85RAaZl;{7xp(1I>Y7xkwvR0fAWcejQ?*Wm(54)|w5xN?jo2c@;0A&Plk5kvN_i$w{OxV) zVpmUk$;a)q8)$qywf&Sdb}TcG6~OZy9bX8FTW2IHW}~imO)XinPKo*kyw;(8-`a+F zv=e?KO{k&Spz*gC0K1-W1~Lk2G`O5bUtHiQ`f#kh+TFT`9}a@}rL=uUXhFNLMa{~W ziXdcuUNL0y{^t8~!>9PUq|@JXExy31`2p$1%I#`P+wDR*(_gmb#-MF4$S4Oo0EoBT|h)EOmPrvq=7I&sQa zi+^yr*fTK&Z4cTt^rhf%O%WZAFXX^?K)A7+va+)&&kHKv=H4$M4ay-I?`U*i!nd76 z^o0Y*IgrOYV+6-W{Ka9m{EwS!XO6~ABI?AG7R40IWIPv}zvHoZ?8(hDJJPwiH6Q<) zBFKHdPzRNXY)Bo~oTziGi@6y$BYFIl`kJ#ozW!SBX#TO;Orddn-@q4a@28Ts) zjzvRqY~wx`S^}@3HV+09G`uS?)BUFI#y!DS8O*{JgrsGaaybVh|Yfe{cTKNm$x z{-B?nJIQ*paOe@RneMrW^<1T0l$?U%`yRjJ0%Qu$D<8OcZcP-$OPDAICb%!!?Efgf1bcfcoyXv9o41@L+iyPd3E6m{!7BF|o8C$+)Y@9_|F0H#-lbYi3#X-^I%)1yC?BKVitGS)ue}F9sWombC1C3dn zI}i)ll+$0C0%Zq=?8;JobG1v&z7CJ`P$9DoByaTB*E{@%et5A0(X=3r-)g!E!Rp?T zwvrGr~$HkwxCsyg%Idc{XZV-J(k$)}yk3cs6iGak^Rw0BMphKtM!7 z_?JtbtrAjWE}}bFwsHFvhC^uV+`BY_y*YmW= z>>vD~PG{DQMwWDuDO1D{)0C)9T{j#|F$s1lB9|1QK~Yyrbtd%$#$*uWIG|V|MIsj-|)%q|vU#&tBz4(~o21G=|m~0%*G7p_U)Rg2ZX^#9TiIWMVpn<)+G>nRbFf za^`)sPtfYdOO%2#j9;{+xdJFD%4#tMC%2pqC}k@ZEHDu2f&|<;zLDsfeY%oVmR~3G zTQkj7`&jA!xcg>IJmI*cvsb=rTcnSwd`hIV%_k)fZ>>DM$(IhNbA(>vjlXEEe>9}k zpOY2-L)VJ*!fsSUv2VOfFT|kl>kR7t&|8(#Y7#bXV<#ib}Ha z8I9s}QRY@=>&ETp-g7jV<=jt;I6zOc8^;FMq){ys*%oq)=28rm-2sNAi|Dkh)+J=~ zAMEII3TDRYh+xj4N4X$ec5=oNy1L)NA7_V-Uv=B9p&3+GtA&8pyL-^-sMNgeSYBvX2mI9ZqUD8Mr{Eedua<&Ysm<}NKM@Mi zJOO7<xA9w^g#4ryj>#G!YSi6TSWO*)Yo8A&V}#CQQ+xVpaiypuDsHBn_YR(F;3 z519VX_<^H)?=oQ+bHiwJC$~ms;`D8j(s_R=eWf{~;cEO_ugd z_K9*d;yPllwl@A4$w?RTl({q|jZqgF&chb>+f`_?<;Mb-8#i9>0=rG0hBZbALM^d^ zV-F6JSh(Io_d+*MfnT~srX^Hm@disOeu`_$+Z)#U*SJk>o%-3lrb3OfyQd!x@#A@? zF!jfrg*~d~{H+_Ba?6Vadlm!5R`vR z_+@hdN;ZXQ$6hk;wvixmUfUmHv^;w=1NkxG0rBvNO3^m;g<{#Tm~0_bINOze!7-me z^9wzdS{9VL9<{H3x&obU3^3`&a3)n>j9v%72wS>d9aml}jAYu0hLL0B;w42&`gt+v zsvZ1#n^JvRRuF(ZdgDMyIG$H5U&~ajpt+YH>h%ybcy+GuKvh=hBo0wU`H`AvF{9R} zO$E?0cZBfO)QDDL1zUHQg|oNaforB6cwu@Fd1q%z)9=4}*c17S>*Jsx;;?nhZJMa` zXX;Kr2fyrSjiqD!?$H6@lTVZZ{hN|T^0>Z$xA()fX0ouN+M`Pvnq*wm&9vLLAzHVp z51HvYhc-Lw@=yj*0KIT1Ce7p2G+>6k7aA|A5;T`%Ly^%O=K&zhA)K9DPDO?=0%tup7x(L;bS&H2E> ze9rj~Ah&e7!JUNFA}r=q4M5zwoIA%PgP}!Z%jUq zC}cb$WIl%C*hOjPl;%>VmswIUo9wLmTrN08mBjY_uHH00d}3#oQA_6P`^}cu>y6w)9X~$UMT79bcau_E$lR?ha%_ z`jrIKiB|LJ`;oZcx=jY^tuvUq`SIudw~lBSq${WlH0xg z-OlEV;WDfK%lqll(!;Xjw{7RQj27858|d$9{HU}z=NTsBgSe#F)M?(mV;!VYcjbNL zUv9s(tUO%J(_JkiZrBfCbN(+&AmK0JP($wrhXL00b)0GS&7a3I66v>xx7xaWweAN*qyCCO1nSNel z`nwdLsLorR$96q&nIB&!&2|W}8M=OO=^eGHD&}aUg75)~XgQJZQ5r7m)u~H=zJP;( zlwakF(CzFd`Wsp&pPQ*60g;l#v2Hw_cb_!WM97Q_5LrC>$iR%wzW{c7-c~xAC>t;%hbkS zY@!m~QQq&ry4{5$G<@@d_JrYa^zPX!)6?3i0j9?g$SCtm3oapK1wb+^CZAfonX#`_ zKSegRLs0_AHl0ePE9}@XGG9-^5xZVFxRtLHa34==PNFz{{b#@c6iQG-*u{Ft`;(kr zi%!)#o%nl$&l%|w@r{?I-{jc-_){k|oAd>l%>y7B&k809wR=u3FBu~yi4cxXPCi6T z66Q5uXwr);zW8VxnV#s=(XWAJ_$yPKu*N{7izeQ_gAQY*(5#RWQlYGXXO?@ROw@hz zO>ZE}v0V8eswkwW!06Y(1}Ylu^TS{JDB2kC_rF4-_-*PT=mBa765n1%2Q)Sf{S!E= zdLma6;be341?#39uo@<=;-V`vN@4%rN2_D`CA3IK!zsu$9oPq?B}D{e(%8o({161i0}N-ObeRw;^#q6!bxW)_qeG*NqQ1KzC7@hF(@-r@x=Q#c1$+hDFp z%ztJ`JFt$Al|sWn=DBNH*;%74<5^kT^~qQcM~nfzBXe~S$(uGN7q77=GMD`gaRr2# zG|Vh(4}jl1rpfk9o_S)jx^eA|7M)0knD)m%+KIS6Z$&N+k7NDaaAwNLj z2AISGAWR5uPWTz*_tFbGYhmor>7OEDM9+VZ%MlsV1f9HDrX?!yr_iFbqp(VLG;dJo zv=Md`OuZYr)4oyADOA>wgLEn7|D+Sml}dd{8ZoO@(vOO3Wy1r1-N{9PZVAJCgn6mx z1cu8(I1^MUXvG%nu@q|J{#~F^4j`02U?s;C8#JA4BtDIeg=xjde$|AF^VUb;*xiagfln+m;o_zG+mQ#?Skc!qC$37 zZR;n#wQ25HeEV5d_r}&IfIYJM<3wVBmYK_kRQ4?i*$_jeB5%cSIm@}YKU0w9HPwCI zVPwf%Ig>s#5*=Y3ODG;RobCCmUJK^hEgc+fKyi&c?3{kvgEeyg?D=lXg$j~SGtL9X zNU^;LK|;|N88(8xxvyvZB*u2!+|Tr${W-hE+Y%A;>tn0gV2%kK$tf9;x;|agNDYY? zXfe)r39ldQJeOr8ybhH80U{KkBHR)v87Ns~naCJ!=pU$0l{A%eW`2mvPEnUP<{PP>jG){SV>7X@yr&b)jz`R!Wm9a zPutnA#V(b`-iwq?GXx&s2PJILRNcm8n>nTZ`QGv>DUjFbQ|-GJsNM`Oaf_BM_kc<( z)fZD;uW(M&uE{N`b&_|0W6Yd;O7V^*pguWg?^_}kkI8;kha$b{$xr#O@!na_e)Ix~ z+ySfyg8cf3S+^J8`DemD6gslu;8>w*)cz+hdxZcZP3YMCG)nTta(&KVDg#ge)#k_1 zLjv0-vIeUqjZ1J0?U$v4G*p5m>L)Ne#QzJ<=Hm6Q=if^fg8SE*WAiq$BzNeYgX`C6 z{bJoTz=ozhg$hguzLOyeLot&<(L4uo$hz6uL_{`-r=cNF;r)JRfAZHGGdXaSRKo%If5KW@JG-t(fo7Yvd6NYD8SAsX5qJTjJEGc6g1_Fv z#r5L8cGcabF0wX0guzPh6_v21;x2?ili)d3MvJNUX6Tk1ktQJ~<8_vB&)b28a8j=I zw(IBY^96(YRLC9pwL_Db@Y+*5-;2U&VEz`~zrX|MGaNbmwApW5#04j$Ste)E7U@Imv@-hpWu&dPQQ4jCe!X% zD0|qLN>eX@vB$VY>`!9t+jf%Fz|@;>?o~N$FR95bcV*7@mJqTHWd%2R7i1g0QvZ7f z0S^oePwb~1A5+x{{yzH@a1);zO7ni37cSD;!ib0)^)I3LGB=z8T2R~BH66tPZ=g51 z3BNpElg9kY{*F>Q{wBq0bT-!k11j}>j~_w{cy$|AcXvWxv;s1{En*+(+CI0J0MqMfF3 z>2;}!sk=}@tWE4JX$QQ*;$nU!2#cw{>lUX3?Qnp9h}&)v{O1MxJW@0?Cl}{-mD&JQ zL?l7EJURQ&mvLA=h^Ko8KW1_A>Uk_qzRvT4Rr;D^(8ANwWFEFFK!JAXuK=R?V~j$D z#Wp_SY$dKU(hqC;g4nD@%V`HTdrHM_mR?FwzV6j zQ>=0IzOD$Pt_VkE(6+Pt&@?JtC4lSDPE5MktSW|56T8#a zC6IwfQu99Cuht&Bfd`V|_A31ACgDuhjj8}lAq7v?$GX>X&La6G`Tog)e^~CwD71_7 zA{4#M^pgiUgJ+&MJor?%SQ4I-B<{X(d8uD!JpMDgGz+!{9y>p(yQb3JzJ`XJ!;(S# z;=H3}`Hy}efFX7<8GSCb~A?Lo3lk%#bBB&Z(V_|QnUBFJ2+F?3pFK; z7q!7C2J&$40?!t?C^B;a(lxce-}8`7TKr9*7Q3Nbo4o~{ z?pbZn)}*FS)LONYup-vV^~|mC$I*GGrqB`6q|-fucL_||5h?FpOg<2F{3P+Z%nX9VzI2+ zqwWpQ_R9po^@-M1pMO^0V7Sy?tk=DfBY-Vd3u8z3zHhY<`T4_G&4k9gIT4BK!Gq8B ze$t`XoQniTv_HI@Nv@Vt&kN&QPgXe>19p^{tPX5J@^fiCc-OoTx0$r19CHjfI2Zwb zbo9UeG4?}590Bj=yzYD3wo#hrNFubijfiE4@hw(~e_er;8Cy7G*z_Hy#5j{W>V$opgG2 z8%`UjHA}#QcpDQ%j0(yM!g*GNo+cC$KApU<%-n zoxa0DHKh{#DZRr;ldD1+C4K+gR89vjOTwnwpc!e6N?i$5du^U=ktvLR?5tB9ian_% zNwcD-;2jr=vXu$*HnE|?w8h~P9VTSw(DzNm4_IjoB7YCh zpQDtqPd2;~M0FwabEsFKcf%c#Pr8L5R;QHA2N60?wq^1w=?5h>l~1)1H#@`FH|UfB zS(Z3G1j^^?LX(%loFEzGCfP55XTToJNb%;}iTR>*2KlixmC?T2>E$=ucQ3L09lr&W zH}CD5L`wTl2U4*)njK2FlouM@iej=ig-Y>fJogDO8%Z9gAq?(<$V9FN#L02Y@RKOy z+Ef0LrYUfq=V3{3Ez+7;L2tImF<)`$ot3O%OcES#a${Z|F3k`!Oc2ML^b=gw%~ll< z4vufZ4^7RdG_PJpFLUQAc_$%*16IcjB;1}{TkQuq(?&|=ge>5B`)s!Dq^2fI2Q%wn zM{o_qH>VWhqFSw8b9^&imP95hnS{l?>GzvZ!pIw-d0Wxe7Xh~O+Klk(yq(4j{(+@@ zr7H3EEz;>+u;jWT4RH*;oz(=bt>F@C46^nfeg-M5g3nmbKyWc0hfIK%P_-~(qIFWN zk%7S>u7DGcs3u0a~y_!o^NkB(N$})@E;5k&L zH|(EW(GLU?CoI7^VCI5G`f10)PaGqaAEGb9t^h#8k>u#L^em}p`f4I10FlLn2^G`4Nqww(rzziFR$ z@BO^rul&kAGxu8SS{IMgfrv{vV&!}a!_I7N#Pzpx_4fWLu%hfHE{so&2Pcw6F9HPR z*H5Y-XD~THD|+6^_`RfiH@z16QJ~(7fb)k6_YV#z*R6g(TlQE`4J8?YLDN&L)Yj3? z{#fs237~T-N!^{+-wDX`;(l!U6}?6AKUW{6R_tp-_`E z0TpdQx(u~tCniZG!`3B7cQ-%AX!FOu7z<>^2Mj8~Ny0^Sjc3j0y7Fl(6X zMS)f-bHiZKkf{v}17*HcebDnVR(|E~6PthINSE5`$uZ|#%jr41;t^wddFBtvK+E}@ z-(A(C>V!#e(TuTV*YnZcIA7y(N=uyFd6WhagixZ!mB>J*S`tbF$Z`Hk7DEZYWpB?M zfW>B&Qlz7fd-28r118$(m5 zt7cU``U*!H7I#k|3(N>CJU}49sTIK->@AdXEMt~UK@g;{l*vPzqk_0 zWH7e+s3N*g#FC?CgJe2BMaDZ5n4u*VC2DjrrIdI4BCRX2F3M-Oh>xLAQ$Ay`XpS7h zR>IVI{A z_n)vcKxjo%4Sx6Vs#~)lGI9`9MwiMb~Jpc!l?d}Mhbe7O|Q~W$R~xz>;r;8 zmNiMhCS`*^3~ekiX^$ldausO!pDV)1zJP>CjhJe5TE-c>_@j_^~B*r zC|YYiK41EZ>3IC$K>&za1r=}SND2xP4MA&Oh5Qh9VQO=KrEYKp^3ICW00)h#?J3TU zt;k26KOyz~4YlABt!RCv`~)0mbr-d$msmjLXUCD6WmqyKV>RP*(bl9P;BBxU|;*HabE^{wTP@8dwMO6qMVUCi+{k9XzphaYE>UptdGZ zXs2|3Y^A_U*~<9KLQh1Zcpx5zoQ`CHUCIH*BbJOB6op zdu|X3ZCY^jY&>4m2blY`Fm=f7olsATYMV#))2RHu^zS!|ZEHW36=d3(Xp^upc=84; zF`IU!12-$D@^dMY)IWbh^9c00?Iufb!C>x%LQMdc+VWIJyk>WERT`Ed(9S%!94C^8 zsi?QqqvJ{LE{8IUUQP)BCJf2P7TcFfE7P6YkOgD6hg4G%4X0zDV5vXg5;XlX!%@R9 z)y{)lII#S>NK@8)t{&D`%zWp)?Z6<0`jZhNy>qj-J^+*Mp5XfMNl8r^87J!X^d-9; zNvchpqSKgI$^BF9pi(Btckp9KTwJU_{Y7K+tSm+jtK;B6lK2iS4E+CCC$N#4jE8mn zN!?&%>9TbhtTPxbFCyj1V}e9}9WaRBHoxAiyB545%l#e24po$AO2$W*JN(zS9XZ_x zJ2C@U+-Vq${c)j~EgXIJPgn4SG8(<1nxOnaHgbm)q0MgTOb?R!;P!_HRw=3u5k(GZ zV@@7~&mNljc(Wos?uriQgtGOxT_cVqog(GRRK@ZHP3}VD7<)w`FN$MUY(#N}k@yk~ zm!`4sB`K9NL4`jwv1vzuNWiNBlp4*t9LY#OxH^%NuNB5j-{f@egYRw;kR+y^C~R4T zMnLG{f9mvP9tAPTz~ZtF}qR{G8nmTk3tUMY!Q68ARD~ zOsJn71Vj5V;2rgXEqQ!Je0Oh+)LJdTl6G5!oatiXKEeIY^VV!666B8cU#6xJ7lO#& zoTur93AubJ_~z2}+Ha8DUJ6D{lf91C{}Z%=*2=(^lk-jXIDJnN6(BJ%zM1wD5gax7 zl*gz(u=R+}@q8sus<1Ddyo41wh*^>5TUjNV^ZSBAL12zrDKs5Jr8~!!_g%lCU?%xB z@AkBr@0?6n&-NpRVi)wJ_M#_G*sLH>p|>xk>c(bpfJX?VUJ^oLh2!)FK=%vPTnZ5O$3Db&)Nfy5hbRkQNF0|sL( z&}py7YY$|5N1#BQRXECgan9=krMNS+R*lxL26_ci9p6#0K9hw65_xdiHEM}SzeoG5 z=K~50cmiVbz=&P1xi_@KTYim@;vQ#NsEL8&m2B(X*|Oyu7sU}i<^Y8Jeqjz3e)_$7 z?1Wux^y85&q$nSeA_$#e)PWZ*(JQ(K*@#>U?MI>oHfT65c^U7GzJDgzLu8-Ki5CD- zG!>@)bagM8&t##ai?{k^%CDy}zm^Zeod@MBK5ruRKAG#-Cu7Qj?O?NW@nk@|Pu(-n z1nS?8Nv$B%GW%4bEergU_j+JsHWOxz|4?ZVf!o(Z8W(L3pJW9CSka$xnXpMFrisB+ zJFfQFWr!j4GnZAIz>tM#;BvdnHZw0%lQ z@ON=+An;46(?!r|EroHNl6BKwUrGTrm)YKFP#slq=zxukXC%`8Ik#ba`K^5LkGJh8=xVWx~ehn<~3c8j_69 zWi!AM*} ze3%Bx#O3nz zPPe?DXEaR&VUZ4ajlL&xdJ^IQq`>#S*dlELeBtA18yL3`PVEE08Fm~1tklPHouJ|T zPVP6N09X0^lHN*st;m(#;|4%p2pFWN-*>mVpjWUNUQ5xxV%{aCepRGPKQ`UOu}o0m zvh-#ri(=`PO*8$%G=(Vxz1+dTN^593jiuR`@a6Gw&XjP`I=6l?TU!Qj0w|y{#G1NP zOgy@X4HDj!G!W0RcM-@|Y*%}7`Ht28YiN}JG!c(FCZTXf9ODtC$~3cOkuY*MtJbWA zU5RvIrKP9zadYD9t{=NYJ+_tk`0x+@T1M8jpR>waIkVJ7P3n7WDcx`Gw*{74h(`k! z#Uf91ovjDEM?BYa(}pY1H9Yr`%=GzO&+%>T%`n%vw7?r67v()Yco`A=ru_}Abe<9X z7tZ{s+TIas!oVe!Ak?Z>>&Y7@gp8x@Ni{8Y7*d2^!`mX66gb!y68K^MU)lv6*ZZjk z3AK==!LvBnnU5hu#U+a<3PU6A83TbiM$6ITOc@SM%*3a{xrLP#)4kg1;~O!xDWbp* zGh%2x?%-L=X`}SPR1uS>q{AjWWX_!~bB_z5#6|f3}_Y=d20*h|Nfh z9*gi1U)&dA!bHuH8X_HUDqXocT^X4D9mzfIMn6F(etDdMiLM>G%wPm@O%V z1wf$D>kiwHq_`Qr;c65CT4f`me#MP&jJV#briAa*dezYrJ|$d zq+FoucU8~jM1}R0NgW)gA8)7>)%_-;U}$dYaToY&-@Om;Xb9RG!Ts>BZ0R8?tg=Uj zmB+6#8v22wwSmfjXAb+kczrbxoCk~J-l!$$Sln09uy*xuqdHI<>ZKjy2vg3bhvXt> zRABAI>d)k+db=k#t&t?YgI{+z4#$Y;8N>E!wbyAPJeoDBbASUG56KIc~po&y!B_GUW0eMQizHD%pGOW4er@fsW z2SxN+Sy@Tw6&k2*|DLL&k*rwTL(Ft3J9GK_&@@MvJTOL9^F2hk)%Gt9-LTz&!m6rx zbUZ#s1os%CRd{h~k^lN>MijPN4X3J+GH!OXsxJqYf#*E%vd(DfPUx_9MTHS0%hsu0 z@HA`pj}tzkUo`=y0$gwYvM|ulhQo=pIIDVDMC-cPEKmz{c^AFv5#Xa&Nk87(pX03Q z9;eSzViJzxG$IIErh@?YS6h^bB-`ncLtG6=4c+ufQxYQwVHBa91sXStifV0a9k=%# zBw@-HNrWlaJ_lNc8lSU^C(4#xSPg~#pYh42Ae-C|{n1^@$KbJtLHQx5C#Mk|nmms9jo4B^jI9*ApqgW&xGz%-1^zd&hh= zSnr5f@;^yLAxJbyl8|Q6# z-4QG3XG926vk{g4mCW|z2z|$GQJ)cyofOKL^@4*g3Aj+UQVlK_W?_GMs4=}GgFKBS zC)BDVDp`L+b~{><-Iz+n{_-9_+PHXZe~LIUHCk{)%b=#~BN76{8NT053Z*o1ReB=O zF-xBcqX2*zU!;>|@8pf-m`OA=@bOlii^Hc0eaN8d%ww8`_uu!gCNqR}uz5IDMHJ)P z9Y=NaKxfi)Ho@*(js_#WH}lS@K=3n5x_S#tiJ(UKXyB~rnG!zEA5Ux(ggPiS-uYuf zjnw%K&g6M)2sz)85bC!lFM3mo5ao_3{P)pN*>kRhSOp6q8!A+CZw|jg5SX>D z7dbts>>BS98w-?my*ru0Oprz1wK1eE5&iv5WD=CVatdQpG!O^+TYK;u0mtg0i*=44 z!EuG3Qdn5`p5{y0+4Z}b&mL8I9^aZ~12MGjdWL#FsPQ|prt@p_(%`dL|NH%Z5`-vw z`~oIGz&$?>y(tjJ8B(Ph2R&ZvkzaB`YdI=dT9exX#T)~UQ%bF%Pij(_mX@5rh z4ylLboj4mTgge>N6$b{)D|j_{o@c`@Vk{ZcaTsnee&0I3h>~j1M&gBeLJkiGYW;hI zjYJ5QM12eeHv}9{cnftMiCQ+M#{!e@hq($l36^%XExN*PJ<#wRAJg%#3;*WElS%x3 z<`q_kV^f~~8rkYCI%j-C%^y$=eWwTG$-jt`Xtb6~f@ZhLH41Q`CoDpg`uC^Zg$R{U zrg2$LrZM`6n%ZKnGPqwYn^P@hL|NH0aC}y>u%gO~H~4dOB*aJsDPTr6eiJsIqBmNc z4Z&-g|A=<~ILl$vZq{^snltgShtb9->U8&Jxp;`<->(wN4a&m%N-wxbcdr ze?28~;9H$f$l&o{)M@y|Cm|6J?%&C`VN`P!(S)j6LJw6dq22i&+w#dy-ELCNr9DF; zWWmN?Z~1>K36mh$=j(knNyMPL2EbBg=G$VL9vIJ0j_ zWAQUW=kyo?h*J4hhU`U8xS6f>DWRKn)1`sGZ+{g?hDF*yQhX!(uY<~gW{xGQWFg9T z_tpyMh8AYaCZnLb(wb;I`_G9Kgs>FQg;$a$PG@%>+ep2cJV&*iTK@c9#b0kjWA%I= zfPwz+h^>xva$8i%&YLXSYCYgmN5VipY?RVc!Lo_XnDFldYd=BNdy9pUM+fz+8KSOP zp75|+o3cGeZWUEY&^@qQKRdd8$7?`T{`a=44-|^PkhL42usuyavn_phO4*~W)$D9s z`uB<298iV<-L|36PKcN+g@sgbwEjXyX!1HZhNp0TZiE(NE~-qpJWksj)Qb}C9M2VV17X*H z()iHc{?F$4-)a*Z5yVGvwxnGgGUKPuy+pi7W)X72`)TLt(lY=2ZhEM@iq*h;iY;4? zlkAz=Me7nBgXJmeygBnt!+DA`S2pL5IxACKu@4CU-lO@jwTl=YriOHPFO9^=d!|zI zYx#xB_=*Egr}}1FKN*aF%4cwBzGG!hm5QLy&$jiF{Yo2cy%LgNxrj{?gZRJo z?>~Xqoxe~7s_etj(xUi}^8Hmqt`kZJ1&^i_W8!~SoE<`9{!F+(3>Pld(n7z0AFJHP zY>JeT4-H1u8cX#5F5N#5O@n>!g;A$;VY-Mw7T>)mcglmPhfsirZT7?6h#Kp!g*%jl z4Fqvahw^`}q~0s}?672`Mg(_^%&W#9WXPkOg6ePy6>Y=qYoI23SgAhP3;v&MJJ?nG zk&h`nPIRlQtDC#KvyI}rG=ELRG9EeOLim7>bclR;1Me#l>oz52>uwUxcmF)C`<#hIGFy9>FGK+eihoSiccIRe z6@q7q|5Twj86c9Aj*8e+$z6MNjr##zI+oO$KWYv>3;f4;D2tFfPB2rR3VxtwdAERp zMk zhoF%3XYzZ*#>V=Fhli^&bKnL3-Md5|{6?^O%$T(*pPuSy#y2<1?LHRyPS5nt^dmY-rtspcJ}_OdB-JBv|IdmQq3Qe60szopca34lJGc)be-5=@ zePrhzMH+*tmChr9_TgtadK+Zxhcrc_tfQ?U^cuK9`qo43-_Ux-%-xo*@(j-hGdN`{ zQ!cNy76j8XGmLn_o16Ng66u`|@}mP_YTBo!w)%D7h#DiUEa4D*pRbzO-g2u8Xj6=!FTKa=@R|GAx+{9d`;t@xeisT9q!X$tXd^4xLkEu&>Sc-W7-x*Xu7^sOw&piw33%a` zsFij8G|;g-Q`wSAc407%G*Mo=#W*d<1{hO8Y%&aJ&NpM%h#> z5W00GMx43mz$_hCxraT~U;(nh$b!p$0vmlWQz8Y7#4QH{K2!$Q*NMR}p8DeJzpjp- zvtOqPMOr<>L((jeN0GHIV&sQGf}hDzO*a4KQW3XHHEoDmE6ip!Lu2d6mcg z*VEM|44?ut+3jwEeD~a3T`|9^^&hSY3<|MwjxQnpAekF@j5J>iU!$N1LR#RDor)l{ z)d>C3Q__&vnhmwC31{%7ignGO$U&Usu6;06TJ&aW=J4Z|_Lx1#@At>9mj*ywR@ia$ zPxB18GI?VWLYm4D!ki?^8#wrIRbO>03vM;hNQp8)Gh*j_!DMQYXt_nIiN<-PlaHwS z@nLf^zYZoUKC@g&rn(yVAJj|;D3Mv5*7cf9<%$hEgo!LcKtievt_33z*n+XvD=Wd8X14!iB{lv$o z9hDqoA{_4ZGbPe+4!fbw=gs@}*Zy}K{}qhtBHD@cS}L%w6!m$t8dX*cyTZRF45;JO zUCFsmSr=l2h#4JAlHs4gi-IsUi%6Zw59zI457h>d&g3#aYkK%O4HGI2$1na_xECrY zmx~oeAf74Lt?S!7W4JGcN%lxiFBLfHbjWS^PP=uLQB%(u>=p@)P;`nS(g#B$;wqfz zr(_?kHaW}*0209BHXeI*H{elJ8c3Tv9YgjRFa(InJRN851xBP&Kn#+<-*k^R-^7HyBG}PvNjm zvj}O+RPVak3_zhPz(_FVI|t)>=}f+Xk;5V(47dN>5+JCw4D)aVWC?LY_-3}r?@|>F zA@UR6mxpMf+c${D;z7Bkwb@eC8*~JDDmWSQLiA{wnod=wBj$%(bv*p5e7qtj`XTB0 zhxj2Uj?S0xhj0~{ZQveq`NrU!TAF0Ti81zLol_Lfj&h604;r*gD)WpwrP2F&^ql0} zxPx5z zh11M$r`x~<$iZyjKr4z-l9W=q4ayMRzWS(MDO42Aby`;6{ZuHQRSrvu+_7WT8zb=e zle}nAFI+{fw6poFeuc@f7gntDiaDLg=IQRtq6Jm9j*NcgPZ(=YK6xrSQ+~0~*)>R8 zd~(07e*qq9j}eFzs4?q~-p%HbhOsDT%c;&KWd81~Z$MfV{68#!fsVH_SX>xRm&mss zqeDIm*l5qIkMPqlRr)F@JR4hF8RMVlU6Y?s^@8DM%dA$PUiA?e^0(>#@N8syIJH^G{nh7Gej+0f?$S5JmmAQlZwS^ta#LE=C_pI3-3?e53CdS!Cf$kk1JWKjz3;*D`=!g zI&;MIa#wnpary4WPaihDdgHAc1cD~o{RV%e4^jqRpny-fK=xb>vRUy;^%wr~bUp}h z8e$HEy=XoSmFb%rDJ=^!-qq0Mf@G+ZB@ZW&6GN%Ruq05-P?Vt=$GyXltkAUle9}hl35ODtpf`jcUURCJ4tutNyPS zY}tKrV&4(f*n?VH!le-~5oJ3QSFNUi8V#1Ygh z_1?$BjP=v=3U7-j!Pg@zn`Jdn&Fa2grBM>&UVCZro6+yv>%a{0ld~1vKI9Fc4Q)h&M()9 z1YF)+&rh?AA}S-1s(*S6{JdWtY5hZM#&tvx+77Te-ut}0+}nYrM)qu!6%-V(iu>~u z$|@g2U(&fOR|_yXIXPRehv`*hFLUesGOtpFwl{F-3AfEyDGqkh)6-{HQ$)8fd5BwT zYHF@stKFEd8dC=iKO{!wH>;CW%l~!B_QkU`Hf}^X&H~Cu5qUw?vnszMsyA+HrhLxp zXn@i^#Lam5Agh{G!*r)RXWRSp@DCjfWb9+=h^aEE6wcOVQ^uyoxbrW2+;wHOnI(Aa z_>F@k;j`Dbelli0=C??cRVn&Plv?~L^%0Rd0xh^AE*w=LO>MWpUIU*ijAI>}(u+3=txJfPk#`3+spL>0w(r0uK21|`LYk4Opit`k6W zP{1_Xby96F9)sO87o%Mwm!7_r4vV)=%Y*H2Htk)I-#NF-^}PF&CfXDVTv@od3~#z& zYI zo$?`?D`!>B%wpi=1;vXgLGd5;#3(b+h~N7i@_PYJ*R)aFy@`YuO2x_6Ix)*{ct4&7 z7F#mVio_HpoUmhj<@COPk6G};XtR}IYq`MmaXx}w_*`iaQC?^@Y%oG4!RuBiyS5fn z=Ix%ybb=6tlanm<_H7FhtxU`K%G^Y{EP(a_$z zxVRKR8M04;i7r@*eYz4KRlqVJ>g-?$t68p*edJh|4sCdzL=5rZ__#P&v2U)CJN=yG z?KccIqptSFW!JZ~tQ&f;k36NT1`MF4If941iMhKV@{$G zFntUaGqX9!c*{pS($v#|Q^uLrJ&0B<)7dUlVkVFoZljfSWwQo7Js_XPw5EvWf5^(` zPFieM=xEC5+3vI4Na(g=OZRMz%~I}AclNerq@xqV$LmeT zO78B`cvk-e@#}-j)$kn5MF)(m2_Vz;W*%WafyiITQX)HL7T9&-+~vojG=HEwwR}La zV<`xGsgqeTvXehf?6QM6B(UHuS2mhYMBbTW3}gbka0n#czsZmZxw2cMe!GRqRxi1F zT<;Dw9{u>9m=>l21a8{ybF0GI0U9SK73QEzf*UhN6o2ErtyVjos6JoC`1FpZvSM-H zZM?mznZ&oeKAv=$Zu8yY=y{@DwEjW{4UPv4{iKrlM#{ydju#by_PZ%`cmsUIC(&Wq z2&OQbpJ*Hl0Jim2kC+njF4yYz68m8aiTkr(ch$aS!MtweKBpA3my)4N%H-%Scqh+F! z=(~`XHgIsXdHS^!*H8=owaN{^%u#P$x24>sYyC8_rmCf-^aI!hh5=St%+c+EWj=TE zx_806rlNkkld=Xn|1c?*yL7|qVZUePEYEAHJ^4LJ@6vyInZWUFT9i4DtY|_(cH&^h zqf?jn=hMDkIuawhE2On*V z3MaI&gs-|Vg3stmdhXc#X1a=EY_nuCY`H#A##UpJ_@FnJpage;3P!VqQwk>MYRX3+XC}Rlb89uP2WKg1xl32 zTgK5NU)2}E$7nnVHJkfr#E?-PRP#Alrlsz{>EBm7U0e3xPE|8Px`B2vLh=D!3GEb5 zLon=CsnmFTl2uPnFOrxaA9#A{tqR?C*^MBr(+h=Z0~X1dn7u0XFD|QuQ3lJt6>F`$ z9?I#JsY^Ngr2wg00*1HyIa-Q1Tr)t)3iRfL*5WH$Q1xyzD$p^kfx( z8_Wrrjsc%3iFsiu9!`D=17+D#F?1hd4ro@i3yaIcSy|$|w-d=zU5%hqargWBdRzwI zX)2a^0SUJU!ppaBN{gL=i5#u1m7(1)>H4I6Y1N5nasbVjtxG-teAMmkMkq)W+_hhA zp8i7`)W$xm-$qM41--JJRR3H`(|VH(v%Gm|GU(r_0Y!KCv@?6&V_O}2UFY2pUIQ~i z{dtpxY*T2#Y_XjbboC8($}yEG`73zvgS@jB_|7EX)G+KNCB@Oe=yEwx9C&H#%+0Z6 z%8rS_-8!cF8yApRoFJ`zZx|F~fKu>?3Fxm56mfYk7pWY|9R(4Xg~F|wIC(XT=*dH;lHz*pV*uZgSW)adK*1qo8`pxq}jEK ztMP=~>RktwQ*jRxXjY|ty=FS$@gFgSFlbZ-ccu;OaMXpOC&ZFLWy|uKvNZZ(2qK%E5 z%o1woGO|!DAXsY(l44MGvr8dCI$d4EdDfC~B65V@?JpqbWdG%VYbN}`>R{HfajEe<`0pA*t-(ZK7iSQm6z9j7(JxuGKiW+F(pgEn#A9%&tp#&n+^Wmj-IUxIt@%sKP9+uQ2w6V|uhaCvZf%2Ld?LXNtOU(RYZ<~>G; zh{Srxe{8x1l)8=y!deSz)9kF5w8uXc861eQTyyEj@PLApSxzKeCva#KFXDd)i_kB4 z2a85*Bt_}q?Al|a?K_?qq6n)^75(Y4e*BvZi-UrqGU8jD2xe9Hy7V)v>-kt?IDZYSwoE$F^gsbN6{Yp@7Ib#Kct-nywj^?qq?n6WCe`=XZ;!DQIRh)R znJJXUE6t>!%9~89> zhO7^M9AWQD$mHXavjQGY?>)b#haxtMkZ$VFlREmK28D8->MoFV#!10L5E1$n$Rr!i z{#5)uqW9{*ZGljY6l8L~+C+yy>;ViDF3LHS@&cEd%&lSBnsSRuD+8ZhuCoixD<2AF|+Jhvpix-~zxg5@{VQm5f1_jcz>U|OH<71w3E^gfEXkJ2u-#4SvVNgqy> zpfNc!G|Li3eDQgq-*!F?xY&HfWwPNQIth=x_@=XanAS2JF&Gire;_P6oll+FweBJ) z$A$2AH2dVzF zI*3Vw-*<7s=tNJqr|7nW?z(rG1C1Zk-t<-}_v6iy^s721n&V{x+8v0g-9 zWO}1I;fp7n_^J=D`*=!m^iImbtpE-RY>1yx0;R*YzAHX1`g;`(G6z$;=c|N9>8+r-LF9!qE3GPfCH$t-!hw zyj$mvI{|>eTS|Aw$zt4aC!hBKF_hPfAUn#}*|DtF+ExCrP-)WmNRijle1eM-`7H@Y z4|t=|g5)vb>(T2QL4tF&tQ}s>44d``=!5~Jz^n>0k<&^Zd?3pPM_}jB8O)Y&U|rx? zT~5NhU&!!1gITk#hzrr@`z|laFfo{RY~$maEo~y%%OjrB^X1#0 zRb~&+r@yz&4r{`@1o)6Xp6c|b9%#4nbX9DC+ZrvV5T$w3l8IwLnzTRpM5-&N8i-6x zCgUU&KBPKo(jW%B?RWds3`-VrD&VGsupHhUEui$JYDn^U|B(qUCIVz|ArOW5RK>)X zSp)LNts{Ci`FNbD_o5)iB^#&XOb7W%jkN*Z@S|8Ii=<56dM-D#%GtZpHz@t(T#Fx zxtRIgV}R@sj1E5v>)VSDtmZSnbK9Go3RX~Z0_*J=%*Oi5*6B$|`l9(RA55?n*VC8s~FAFBeC0oPRix15$x-}2ccFy=AKwx$S)P7 z>~APy*c7c2H(je8t1;&ykF%gWBkRZO6%}yt56Ud5L{~lDi^Y|bxcy%9=J1@6{%Ec% zZo(Sd?``#&0lpgp+)ZOu;}GlT!T@y*G_gIJ!Sd03tfZ*}x1G@2XQO|WbMC`(>FP7eL>#~(&2MBCel*$tUM3S zXA~Q_jiGgcfPK)X$WE{*UZ|iTH_jR)58RNWL5i_Ffe7g*a{TvcHM&mLR=-(RkwbkG z`hCF@P74Y`E<%LH!{Ae3YXC)Ade^-;NDMY_hw89w+V;vY0!^6LCbZNh<5Z70OrLw& zC4r;L)P~20?elA%o>Xzvsy)YP`O;-D+WHg;>~(R#xsqRyG#J{1{?4C`G1eWJ)`p0^)eZw~#)8ZvEM%lp+J7Z7%GY$={g5`e4n6-R2hD~(mK z8r6tjx&{bdCdUM1#f+_VrE>y6783eSZcRPzkj-rztHnT-Hw zP^B~))G|DTXWKP|)aq4eJ&Q43;omLrpELduY?IMt)1wL+8%6m;zYtKES)jo#s}%z6 zsXh>i_a4jIo-8+M01h^f{ATk8&|1tmhh-x0*FyZdk&@ga6ZL^VrHaBj{}f=2LR>c6 zdChXP)H>35_ZUBCBT?A851?@=HUAb5)m^zPH952_;phyz;#FEEFV_FJ_3A$} z;zBj#lleIgZeNy|l2wHyp!^fRiDYw8BeV$omd_K$Ibl6kDgkeev<*cKX=zFjm9}&h zg_lq zxuQ%rlJ;pfs^k{OmB4VI+w*S2;1 zk>bf%ncwc`-eGc8sHo?Bk5@%%iw1Q)Q==%pZgaN`&^c(7jH#AEHRpl)-TaY}Q3BkG zIm1r(@xNN@#!=1f!F8L;n^j8FGenkDxdbqU$K}J#wnmqMD+u$C9%gV%A@WW?qQCAe z&|amM6>3{fv{Pe)?G?EG+QziI{C&!0cn(|`0BGQ?Y}RdWufetRL;KdGW@@drwHR)W zo-ph8-)=gTH3e3B;<)%Y9&v83e!_p|WXuPjPBaX{`Aws}cmICTJn#^*&H38tdQcd6 z{wsBcvI$+AfnR$DItygXj6)>ISpkURMp88yHr`D0Gcrki!b>~AesME%%lh~Awmqi@ zD)DNPc)#g1T`9}*p1>S)gX=)F6dngy9ZqzX1)lNn$Xr^qPz2yQhua{*k0E9ky6VjL;E71_ z&KeogbIq3P)--I;fHEfb)^4>P+Vu_!%{VW;wTs<-&uvyaOxEX@99G*F9r#VhaFv(s;Ub_`};evZl8t8d!XDrdX*=d$V z&d8nP&Pw!J6J4GZp9>;?vhZO(F@J|Qa?NaQFG&79*rm2;#FG~m0*;y@hclJH8cN{c z%w_UEEw$3YiOlebty(Uqr$mqOH!_|tu^O;@bV09wK3P}9)Cw0-F4{JqA(e-hH>kX8 z>OHi?fT^OuTIHdU>+O-B-nl2#mnmus{_i~G{e-J)2DY}McH-D*r>gQIFpMr-PY;HJ z8Narcs8L74NG&hK3Ry|@Z*ZZ;Q~W9PyrXi{5!v^`Vg&u-?im6TDmd+Pw_}MN!?A;( zHw7(lTitJak@VVQ*kfMdq^M(1hkjD<@#|k~N5+?GCt_h1IappCBTCrF-!wS`z|q;| z3Whv6+3vgHBX7Zjj<6H5*T#YK5q#9h}j9v`v7g4eydCn-^S=j}I> zbly`eDpq|lrAck*b0fXb3+ zWR^rNxJwNGhan|)=6R^DuSt#*24i9;1caB@C%&Hz{+MvgoV1&uAiv(Dee=)T;(xr( zYOg-*ePxI4b{v0Ipvj>ROXt$`$BZHLsG6U{xh}=cAN_ptdimZ^K(h3M1sfg!0=buh zzMXgeB5(J25jC*6ct5h8!}@~o!*3V%n*VGLNvLW|s)CsbL?l}D8f#(qzRCj zwWhf4%%FqbGM`An{I6(k!TX|p+UGj#%eFL|YJWUVwja`#QU#S{tk9s^Y%aV_W04-0t(MwLVWCsfNMYVeP+JFTWg;EKb%l}Br^r! z#QJ%$Hy7fJj7ZsXx}*Z|z^X7Y8eKQnC3ihy)3NA#hTZYeUv{7HVlcb`LxGSCr!%sY z=HBx?g=){o$8sua{M<>wD(#nZ-UsZH??$r4CT+G<1)0!MA{G`LlF;Yi)Y6S&<|Tts1pHMTA6^|HMwV0G z6k1k-Q#GVAcz?iv7nm8Y#fI{6%kDER;bAOdj86|3bU&n=c*$N%0v?vI(coy2mN$V6 zzK)&CW6Nd5+mr>?7m;AyYjSEl4nb{;r|VU8<+d+s4iBha$xWwFhb%7O!O@6J4V@aa zrW0WhbJIVKmTUL-S(;8`jgpbV+1&o>)5!TxG0A@GNK9bIkwDmB>h(f9=%WS0lh>oR zV%w3=?nhcYS|7TwuySXD3|?_>WVq2Sf2qHqUQE>xJb`yqR+w2)Xx6|OtDa^+3yxa zC4?OC6h2zvdJ6DN97BI425q|G2d_G$voj@NzDw~4tag23!!XTP9I1W;s+5MBKJ&Yw znm~&p5cAQ>UtW1nIh~mz5&LWhoosE$XC+E+%v6A~qWpH+KH3hSbqvf@O|2lOH9@74 zl(gJQm&cXJcPUCJ7zBng(Ysrcu=QT$*Y8igry5qi=ek7(Hdz(pI)$P-zjc(q?tcG2 z-03`W5a~A&OX7yUiXYX}P)iqelwW(v7iWlPn=WVdNJgQ;{kB7T=o)jjN6j~W4O#fgvr4!+8L(1>168W9*_&aluWrXjIoii4@Kma6S+|mG8z89mv z3I?SIk8$7uJP-y}DL(z^3`5FqcoAA*-<{zH*lI-G&A)dbQ?fqgzazIg5^#IW!pLRuL4ALx(8(1L@pRj>7`uzh zI!G`#gY#Ygf~?eJ%cmrUu2y^2?_aWoy)g9mFX55cXo^t}1R{4@*TM+aA*Hu{EzHD8 zw6O^9{*vsmnT$@t9vv#)sl4_PbdllBc^Ra=DZ;F1(HKd5ZA}+o1Ts1=(fxrlK4!Go zCt~*VvwJ%y2tfLgunXA7OD{GlYO67;GO^1wofofp#NB;44VR?A+S^@0T~p1z=-`eO zOAf*h;3`qNPq3#en}=vy|M0Fo2qfs;rB;fTuX|lF7>*)=Iv^w!O=l=t)*~&1V*gX; zdTk-Ob=IV#mfiKtX+%;Cl!7<{mxsq%d-=VU3z-wbR8+awZ;@?NbY*UE&UjG{uE&|K zsVGL8+W{&&G^#Jj^s6k12fe86@I6nnwWjTDOLZ5Jf*AAB7Bp7kAmwvl9_4-Z-8rki z2yME@=EtuuP6c1{y2s>|r4PM{#+jc9;pqKdwkOP9b$1NwuQRUpfE8m2bB}lW?S+s0 z#5>&sa11^e`7B_CD`dB}3x#a;Pq&ctu79si=xTI)Bk~HJ2?|vga5pwsgwLBBXMcAm z@_dUb&Y~3*FoY(owzekUmWdP0f&CD++JOH`DIO#|bv+mmW#q;C3gbLuR4`qCA8ha9 zLN-G~T%>lXl9W?pXAnTo$ND{|yMx3IfF^Xkp>ch$*j%d^f|;W(U0TIe zI9Jnk@z7xr=3!zgFDiyKe}Yf{8YK2&(C3thbljCf{M$S;+n}G5>}pX z_ux9(Khx`Hc|2V-c^27ss<=4VSiUJnPXe*DTut3Oa%@-%+*_EYVoM3OA3AJTa8u&`FtA%70LJ6tG8zU6GHZUI2c!nroZww3 z`*zpl#NaQK@L!w7GHaACZ)~uwLGL3$Vv{zj-Uk=dAjCif@7~+`^#A=J3X7KQpENhj zUz%UD9D=pd_SUZX;Dl)}XF5A_fERf~2Fv*FPXp}G$a%bspvC&(3z@K&4}Bl8kSIB` zW>|BoAO23hTb*wW(9R#O`sxzcZoh(h^Wb<&2qnmjjBP*niSjtGf$KeHCFOAy^Dez% zI83@QnAsp8^cv)J!ePKz(0RoHA(u;|_LlYWJ+E%N7kxsx@ZY}jg&EgZneTmDt?+eu zdKOfbkl4LAl6=Mw>W+EDt-x@~;PK zb_l%i^Em1*#!hU{imr4d6Gh!~U~C+|2TC`E^k5Tk@1UZt7epx=00!~;Tg3_%2ZQ}f zj=+*y_~Ls+dXb$+GKSroT$nC}+w%!{b~j#2*PU!owKgO(5b{TzdOd0JKM)>WAAQ}FA3`^56h}WkR|)K&OLWB^ znb7vHyoGpHITLypZXc;n^50#JA#5zKPfWLWPo5?78c%*b|A|cws5q~i3(JRN2d!!U zU0pG1_17Ps9-v)4jhMxS$)xKIZIj*6KR2AHB>UhX#0M~Nya}X7Zpu=59Ctj+VW~VZ z&ZF*t!wOy;Lz411gGPryetP%NX#b?Bu6x%53?drd;IWmb(vTZ^L}f?$g&#Nz2raFP z=7?V8YYRE7?tV&J?y|Z0DHspqZAyQbD`x|nwYcvF&@uF1$iZvPp@K@IWj(p+Ne7O7 ze2QP_d{P$9&xVA>vN%sZ=>+x?Vd|oZ@(gxs~I>3sWg2H}|!V-ToVo%_^{@XeI>~ju_ zdq_waO%9GHZjI*w$K!`>We#SlpSCTK_CoT$vsH=M5%r_I=UGt(f`pz@#@kiIYC|j% z%(Wo%>&MG89z^Svb~vD3bmSBY`y8WKN&=oEh9uWJ@YAuj*xgxKeN~MGy~BltvKRrS z-cVpQ#renWPIgfNcDPJ;D8J#JXAFi+(d=IDnjDwK1_m0f7>R8+N(b92wZk+&Si0@Z zXITA)e?S#l)z|(uC36iVvUoFfKzf?P-6L`1wHjKHT6Y=T^t6+$qY{zb5d?Xt-tbbJ zDgHnj#kFdrFtay6r3eLTvFhV z|F@CzbBW}chm)%mGvARFF#Pa*LKu>k{IcXp(m`^2OjXs2924AQ=HWgb8jXV)^=~Cvi`wEBp31Bbjx+)wofBncJ2NUZw?F`_*dDQ_`^@z%kc{~onJ39LoxF`<>XcQjX4iJeGb^w2N& z^H*7`tPEq65h1i%$fy9W-^_*_BiJ`_!e>HB7vDYDU;yV3s=D;}uue*d6~b~auoDyS zOS)~@MIvI09JuZvx;S=q|-w&VFMpCL<(Pmjd>dc{il>cToK;;tkO7*~04^0_<>7MG;Q{GZM;aa^L*S@Ogi z0Mqvq_YM^W4LZLkF_XXjgv*UE9vPy!%#MH-W)&upu***0&LN<&_3=G7{APYYOPJA~ z=M(?nTG*MO?k|j0=*l@8>N6I_puCJYWm{baD|J%O%1X8HZxiXn(}XMMC_tq#yeqP&hh?}o>E{|mXFYA3FhSyCoG4l zR_<8}!4pT2SGr|Df6wN-t~)V<#1 z1=O`{ILZrKbaA~JO1(bn?cjalHMnt@p~v|3^=FYM6DGoM`ae_3u#{c13vmNPp16#< zZXqwb!+G2=@saBt=;;i*PV~GuBI(#@1&iG-Xc9h@!jeX&G6~Aj&Xkv9_5ZfFf6$gD zQ84T$W>rWKp~pq}LUY%93-f^*P~~WcvEL)%3t?NmIPT+3^OA@G#RptNh__js9amQT z%F>j4p<7?>1f!|d?47rh6tI&A*GArU#!Tz`z12!XaHYjZpyv*b%uBLG9$ivWWqnG?94si2)x~3_ ztc-0l4QIp@TD?9eL@+&l0f6{3-uGln?;BB@8kewx8xU?O`-2P3*Y}e*3VZGg5NIr! zDav2?>^D*`8#yy*Kow9-S~Wa6w+tC4i7$;!9?lX1AxoAE^@9W%0)hgPF|Wk~D|;jC zt4_boe9m+XA$CAU;k$O#Fa6;OOu8q0)86Zs>-kNPj#1?VqL)2&$6s$(@4pVF zU(tHDpr==+ga~O(vNT>{UAkT|V!RsDpsK~9I==9ovVWoU6onna)slSHA@NOi!ncp# zF2+878sI_Rb7OZ9ST1#0JtrpsXp%6H=Z1s`Xlrh~!@{f1f-$x5H}AeKvcbKSOqWM; z+d8R{Q3Rm;k)W<*;sv5pK$OpHG}lrtms>Zgy_6`GC~2+p#o|H0BhXzDk;KHHhPKGJ zIj(;ZNm!A4==cmyZhGpzZ;vM*pc`)&a^Ae}S zMw$yV^l*k(UEdMIb`_kTBFji#3aLlWCHABW-P(b=q`9T89u#uuN;us;=CcJ3=kue) zBK92f@VKIM6`ix}2v)bvamS9%>TX6>)%oL`_1H?F1J**v#hGUddluZ8a9A~i{Y%jJ zY9>AQWgvaS2?lzWMgD2}x7=%Zb&ZyFq>a!d%@Ub6WhcHCa;zIy%acBGMbKyY^+8EL zn>wyr^1fygeLJPO(RY_}E9BVFcV?m84aA7wK zsLgYK8yye!fA-4?~gG?4ChaylJD;Ak{n>W)mUi(`ltBd zo_KReo3aIo+YuaQU6j?7h;Pg3xVHvQ^1(hrUa~gR3?z=LqYf5Fn2DT&|6@x;vnJ3d zs%FHlj2H+>ZnJ1(V=R93%eBgM@mvDGRaiOUE=wEA4vR~bS2rhy!zVvVfBJ_tZZEc` zrKOWty|}f7gqB6B#zZ4?+Mw-yt_NH=P1v~>|6~a3S5jn88S=x-Rg!Ib_Pr-z?S7Bx zIJsi?Vi%DUDWodS@>8K73ZK*8Egqtm!Q>{>TOKzEmoax$bf>t7C1!%b1=bO_)lO9Bbqz_JjvXwYaG+>8>%)d@Vq>9WR+Ik-wc*G45K?rD1hBrwb}&X^I?dz>l`SdB%fWOCN<93KncZeDt#E`K3*@1-movJ($CcZ86b*C=6WYc zg`oetWf^3AVjXM6mq!#@9upB?nV|rCAYGeW-E5)|k}8_Aq)5nQ1u3wC;Zj2TzaFI~ za1aml45_g@{z@$7^hKp}UiL<7664ACWIB^>hTFui!2)xpxXgTS7;_uzz_QHJa;e|p zT+h2N^U2LEs_vj%rm}Eeui^(O#6RQzN3~t&6cS8iZASJTR=Zw2|1&kYZ{yU|^}a+?-%SLBsVp z^TTOV*ddg5pw>;M6b_0l)8RZm)g9u~qDXpaMq4-tmS!VAF&aHXp*1zrYx`O4|k#hB~tl(XNZKr_vjmeRZ(Q$4S{@UH>y$Q|R=9)UGY zYuLd{l_a&(#B}uZh^Et7`Gn8pP;hXhf7tFT(QCXE1Wb#ImztH$QD4{jLAqqI@1H$9 z-XR)P)+6KQ1{qwYHBTJhe9$vTQ}!_0kR1wMBqlU z&s7Y*Qu+Gk*ZhU5aFzlgryoYX*Pd%MP|mh{3Q@rMo*PRn-$2LMswuBjIRB+1iKlgc zD%fA$J^6YmB&`Wf?0gb-${6ae4zzQ4o<=lBTL@#?Zco_Ioc|k`1kZv#ib*e68){N;{K0f` zJ*L#Y2Of=8f?qXMqZ12EkqBD_HJ!*>J3fVDsjpX6d&VVjNSv>n~%D%W*hh}}<7WA%u64i8sl#goR5u3vMB$*xF8d>?ed5OHM83lYtx zIxDPq&jfm6rwcMcJbme-KDTk$??=g#-_jTO8Xb=3XhC>+3Qk2f_biIBc@GskPA;## z1BhyQE9bRCrQ2kTclRka4#AVX?|XuErI-PbBYF>(wF^^gDlbprBBBw!@Ll{36^b&bCJpJHsAjS8* zV0btyZr+n9A{=_&awS-<(bmkT{oFe|4E_!Zu~VF?BOnH)PAxG9>o4Fi*ld<}j*IB0 zJ*~p~0oSmfE3&xNnm{;*Oh?pz#y(y%66&i}rPU8}*QD}GLab`nZs1&n`xcGd_mv?rA06~x?|W@r@BF{zFvop zc{6KbrOkKmQU-$0SGLszSj|Q4g*oGO_%1oJF7{3z>gC>7Y<5t5+M$5+QiHSF?ed+S zvlm%M$xUH#il6sdHYQXt@_W#dq7*}$zHA-9WpwLDLs++(uKG(%GjpJ^TBO;$VcogD zv1RHDv+v=a&EE0D7!@{VtgWHn@!JC@5s=h&vKn!NwsP( z`4WJb>=Sk$u+?=HRktsjTqelZhgO37oh>_{Rwx=eq>a-5 zof-noeI57v2aL24Lmy^fF&gQ8gn6Qlzx@Rnofm^fZUTxg%pX%8|9XOM(BoNt#`kOY z;tpiS8H%0QW0U=hnyEA5j9SelW$JGAc}1l^*o+HiW6FdElW31gESWiVHJ-P+R60xydI#{Dg|>sB)ATcwYIs zO%fe0PnZ!H3f_Lt2_uJggetqcNQ4N8ve;j&Ue7tF*U^E#V*&^~) z?6se2)xbJ1)t|(q+u1#m>~zjHiWeoxNwF(9FTP{JC7l#4!)D$FLf4&%(C$>4}OhTu7{CbEs z?6VgWfhgt|#C6zD;5spZ`TRh!PP-6xg$7vX9LnL`FL%7oV_*2(GW>}p+QT7nQaqk8 zKo{J)P+(({7s=@ps2w77sW-BOeQrZsoo($|Z&BIX52a044-{i$Qu&#`gRn;!tOp%s z(dkN3&QD%n=m>x&q#8Vd-M__=oU*#?9jOUgc2$D-y)C2!JOQ@kvMRb3`{OukT<(d^ zjTCgVfdv;)mG)50)l1jDsBd^HXk2lc>uv7zI_@7#RzT%&9{$&3M^_@bs`N8R^%ji( zDU^WTph5e+P_eU@rn!;IzfyKTw~4lNm9w))7rGLlkd~m{-U9+9c21t~q6 ziv}B(Cay}}n2$~c-EasYMu3|Bqo)MJ#6)dDfr$6^INVp7E3g?dwfcEVaTeZkmIyNB z5#&^yhO3XZ$nxFZvvpPrrkRaxGHQMI(yb;0Dbsoak^&Iqupt_lyz_k^j4NA!#L-Ml5uM0v7w^+>KYvM3- zT8K6aCpW7IMO#X`X4%fFooR)SNMQ_VEpA$7d?mMQTjFFlTSjjr-R6Im<=!1jZrS~L z0_RiMtzTr$p+>L5%bL57=|mQ=amn2FBp6k#o@G&fs9$^@$%Ft=mkF^}Nhj z#u9VW)D5(R02JafLl!0L>+H{=MQ8qoTu7{AfL!JP88+S_sygoAs1!0op4Z)wnwr)s zGcJfC-+;NwYz_3uQo;Dh9Kr~O$!wf*6tCnfMOuY+hus2Ce2pca*Y2OoB8ptm5sk}e zOKCPVnu@qR`O=cgsKB>!t==MROG1p`)k{fLH?OS2tG!dfUz}5zOY1mzJ`$cBpFth7 zDqq^w1(h24d>G|6AGe|%>Qw!?UtV~nt+q-AiJE2<5xdBKfxQ5w6mi){@#XK(p{~Tc zV{}(dRk};;GsOxmc?~j;5BWhh_*Rr7p;*ODM{o`RkKJ^x+s#C#n}G}2n_i9;mrNL# zhvC3R8zD%(ScvE_hJ`8|rB+&{ii?Z|G{?8kQRRu`1FtStwa*wH(Zy)>rAhzd6I}dY zU%)q1 zuy7=X{{{z8+y}V%zUJ)C_kwCvAhEZ%SA#XZiWCA7Uy9f||Lp#vUC8Tgj>353PiP3T zSpF(^&FnUK8>dW)ilbcB^$iul`un#Alft5U(8+QBYMXOXUUe7yT16!W3ZDhabU6WB<)YuxO1)Lt zOKVrky~E+?0-T%g-H=AT=Xdr7%i+`1B92JDW9hf8k~=!E|6n@)tg`&I0*Tp`B$Jm_ z@)`2C2KC1BRr|y4j_?`uC)CeVm^aQe!3Pm`Ubhkt?hY8_Q}28Sq`E3scrAbNsaTha z1xKkP#20IL>5tCYCo41OP~AOUF(H&8E>OmG!`ZsD)qVl(o-r@wx-&?i3m52M&cFpHfEpN7%;djIra$ zrh`!qTJlVnv%rVO3RzVrWu%0g>b^m3(*d;cd?#R~v{D0oZC@0v5tR>8I zgGJmq)Zb^eqeW68?y|}H8RqdtP$IxXMr?6B!)9%_ce_KV2-X6)SN{%TVJxc#lW&#K zu&=Fa7*2c6?2$#--fl{Mg$DOl%~h*hAMYpL9G7gJY;d9Q>S(Z4z~GFfV#xa0zFBnK zR+_szCvv*q6~>jIW=hv2`qs}_!m-JX^-7zu%{?mS<(D;+lOzC zIIsUXystMWEgxbHX zP+A;lfT)@!X=<8frCtu!3rU^z2D-25r!Vi7vNz8wxha<+4T*X9IR`2i)%M#Fr$+-Z zLFD%qLFC$hXxLvpI`=c%8*ld-UH1rTC;KCmGnXuKrq7%=?6%TAiJ4FRnipJSs)vGw zNfBc!YLdFLbeEF5J)enzDn`}x^i=jyCGv$L%BsRC(BFyjAVHctE>H6jZ3MDG|Fl=K z^{A6L-PsKcB;QZaWK>u-2KHh?Je?!3VV0h$;8>}P*H0IW2jU+n@!(@u8Jep>H5r0@ zygdeTkclJ^|BsLTQ~Xgh{A6fhgdE+XRAdiwXhfRT?rL=c4H!ruW_Rm|Gw*e-&XO8a z2o5Z#d)>vmSal9EVE2xS)_n3aG_2Zg0}hKzZ4P#u(;r#lHCf38^9!ZMq&XiO6%|`5 z&Z@5EZ3fV3H)Dt$Vl2JPZGBjnu)DYdXFW1!?-&-^=sH(<05^ka9FFAE!^3;!q9seE z;%hC~AgQTe{q~*3Kxs-@znSVR_vx)E=zHE*3Y~&b!kKPv(P1+xK91#<(!cB|g&~iW zy{kL9uy^xuDOKoh?4RC{iPA?<#6j!ygK z;s+(I%Fw(j%hM4|OvP9vv(Lm>*=hF!7q@EN2(vX160t(csY8an?72ibQ;C%W2M0$g zow1nahRqfg3CRx<-|ZVvmo4dIT4~tzY(2Swp`)FWy7XIv#7>&N0=DSR8ZDPxh69VA z!@fr1xP>7q3k_1 zR%HhA;K)XlyU3Yzh5tOE!0?#__CUpa+lQB(jbv*RHBapEcQp2{2+AKD3QazQSaU+j zOq~wN4L(z)GmXATH1;G=e$dl!ZqJ!rRDDuGfoW&&LhkaZq3#f=>>RcC?)RW2j(03R zPts2%p*6DOsm##M>mHn^r>6?+QSA(-&?@*8n%}K9Tfe+sZeXVB^PK+Bw&n?jBA}<7 z_uS-H)fg@>#XnboNm06aQbttPnC(kW(pv4D?8XKFXpBflTf^7ZJKP4=|AL!Co60Gr zZIB-+<%-`7HrRCcwVe45~?`pO8o4%oYhKdLI#Qz9h{tmgTq{{9<7s zTRHRUAhFfnx0y7Hj*6NyA7F>`dK0&L?~S!T8P)yuR=qOoJ`qGU)$qsh=vAny?OL1^ zcU#fe+mY@UOJh)AB%%Y8Vti^QMDKToA9u0Gb@%t}t1P}gjtJH2Z*)YFKtRW*s^*5n ze03*hwc|>XM!g03%j;6HYNHM|n+tqkt=#^&^Xx6r_+NdK|M_3Q$^Qo%*~={sBompI zhpO5`w4T`&brK^A9U&C~g=E#YwfiydmAf2f%{BJCnL)&zrzJiTw}ZMAP-R3C$`ajh zO;O!d`4$06fxLRK7u&4rl+}pvecH>_deGJ_7aW_8Q*s)eE;W{!^kblFlxXc^Z{#l4 z@_l!Nbxdk$tSZ9SCZ2s9$+WY%XUzh5M?Yyf37h>1lk6)dp;)5M=1oOIAIuL#w9$GX z+XUrYA*l1W_>su}W44mO1n`Oz(sh@0M&@%alcm#$+%G8FS818#?_twErGPJ?wn%hx z{fG*&zq8G!2T1d$b!B8^{9hY^I!Wh$jNQkc3zrr9rYHTqp99?uHWF4&K`zX_a5RsQ z`hA_L2uDRiUeO*$XQX(*LBqw^e#zM){E1Ind#m8idT|-kHNR4~hu%iARcnQ=H`O+? z_k4ZDTGO4!c0r0P7sQ`S@dC#)wIot=PD(M^=8@m;3xR!E?DctK>gj3=2bebu;kj=5 zUv9LfR-06o{0@ZBf4W%PsrseWhPM@|t@qN1Hi%N!)FePh5V#I)p{NM!7UJx0S*kZw zqZ@j$7?~75Ci0t(c24gxlAh4!&obi)K)_a^uYU=0@auz7Yfzx>;d9jPc1VByy7FPiy;iS8 zWR6Fbp-opzE1s=cqB5j!>>mMgc7t-Gt#BT28W^D;>e|sp|G`DLbpjfHICK%U$x(W0Pf%!P+rnr z9=|GEYu52?%MKRt_h!i@R5m-d@Y>ocp!Yf6?S;Sb%Z7tDvAiAYf-)zeJXpOkRup3P z4jQP!Y4~`y)QrH)%#4=FW;=5DC)NIBmgwO4xDY59N%g`8E7P%ec^R!-sdCyg;fab= zXJ03*64~I&a#5r$z(subR4%se`=fNlx}v#J0fjyW@PNZzFra_DU)^Gw%JOyMbMo4! zeG?IpSLKaOO28T8y-m}18htLW78jV;BRKb!E^2?$sk?)X9}=cXVQ|jn_)j9zzrF@8 znB)EZcOk?a!wLxHsA0Y5=L-AVyTVe6?~KpM{p<((`?0Q(^kHnA{h;3I9$d?(_Un%0 zL~!ls6w};uI;5TiG~WQZmS}zWWc_Y)Yh2qgA&Sd|@J*OJd#@!M%F{)KbBwhyXv>&$ zo?|WAf4LBpf5ks9tvx1hKVDqhJG;a^l{&ccW$$?E5BZcIh1fLJcaDtlDL)`d8f?Nk z-d&0)vPbiUBWIsaLy>=35u&(WiORf@b#&1TN9p62BofJLdnZ(H-_rjvE}EetzBKlb@g;{Y@%HaC$MB z*2!sWaU!VIrdD}OFg7AzA!~l75h{~ohkD&eknDD-Zmy0Vb@fA=+p4=~!9Dy+KP@qu zt}a`=4HV7pkHzo)B#=aUFZ?XWZWAv?LUc&9`Dn<3qV#O+?2#;rlB)#P)^Jv@2Uk}y z`h)8!o#vL?vQKnLGzEC=)9UlLQ_YT0|MZm0O${rrcw3k|46KVT;!Yf2rcf!ZDYJzOpNqsNjB9@YSl2&UYqkt+{=!dvN+>T8~@F(VwpcHhK3MyM3y4|XB1e<9O zEBOo*kpDym{tY5Mq$Q#vSJJ1C`sAPzDPrg0s9x}2hWVyPKum+uCU|#N*nP9bTI~aH z&))4r@&xW3>m$n-FuPm0b8>~K8*yHc4|@%N-4$FFi8<#y5dE;&hI+R!UvD?>q!q)5 z@LUJA3Hb23k^arU?f;V;Gho(!UoI&z^0BrjMzEJIq|udd#229Z@^}vIXj%O(6l|~b zmVk{JX;_&;#K)5?R0OT|e%EVfH4O96agW~?5<>Vf=}yySjopw)uOQ(fx!GzAx`q2f zd^D|oX3oMkTA=Y1OnUKg$At?xq&P4sRz53s@3~vh`=ax=yUg>p*yb*b(*bsZ%_Kmk zd2PPI5@{69lw5R-zcUg=s%VNLqL}^s<{_`I+{!4yb&a=-S2B!YVfDnzJ8`$zC<+#F z(m!-dMy+pt9k%m(Fj=hWx$?U1bn976kgDI00$=nR@|q-n5c5c-#n_Q4zh{rtYA~0+ zHzWzRJ}mwB#;hPx3%ADYtm0Uk6Tbe6bl=#DX(4EPT+T*AbroH1d20_3WG99fkP51` z+3do?!+#s9nrgFp`*_^ub@7AT>y;>mRy!tn=3`%ihvdMdbZ!upT5XLZt~e2Esf&~a?3K0*OSxmY_i&A0{J_PF$AJ)XrKWVVKa>Qc*Q+SRts zvcqBFSN3)jT5b{&WNr^AbwYRr8w=l_7YGHJtO9F^@pbcz4)@iM@m4W$?qZslh6I$l zugc&P21^Lz{nO~C#?n)9mtP1z&rH)*Uy@6b-FDa%s@2AikSQ1l4??SL>-CE*v$eLe zYIBKq)wbSZ#7ZISU2W_HncDy)lE2>RZ`aD?%;<^33-!sp``5sRQ1u$Q=8TI?yD+#N z!b3=8tX3Ht){Dx9{Jo|~N=mHqf&~ZbS07&YmYjveXtxU?bG>r~Nw{dn05KV^a>)!s z|6A|Pk6eWbbQlI{49R1*uV_$7%=pKOVh8g-+`ivS*{Vw0LW_$Q)B~Ex$#~P{qXbo(fQR{FNkkkY>6l= zxWXz!%-kHRP3wsEA4kOI2i{O(doqWYm$%*LJLdjI)E z@v0$vw>hY`o@-!pCDXKgRAD1mYu+zDQELO?{>oQ!B8%nA6PMAcb&H3s#q|iVpQs(H zE#%MOijNZ=SwWjoXxa;_Pp7z#d*H~$s%I(H#fXatUzO!`?8Z{{(W?rSyJI?Nm>5ac z9zLTk=rn5~BURLEm~_Oieezo?7`HlQlqGEntakQ}M5_FO05HuVZeyfq3OS9ekoifx zXj?H2Gh{F=Z*I9w8LeZf6L#yvJpp1x-aEx+L@=rgFckXN%>>6=hqXPL23{UXMDl68 zMGb6p%p;QBS(~&q4Z}^yuVB>nVofpsWW z{Ov@kpM-ND3r31IQ;FR#Wf5tl0Tx<63L2Xsz(xsPW z)E7fk>Zm6EP;(t8+qMB-ivSK=rY(K*XsKnP^c}{RyWM_TV8F!_75{$%0O!)Y?d)bt zllzQXbyBe0mi0r?8H(06TvLt$3=ylODqT$q2AAPmz3+qYQLqpi`oX-SE{cuVw)!K~ zu+w_`Z)rc;=qfzzoeR{UDA;2pW)~%EX?u-7 z#!ZeW=1D&h?+wi9EQkn+;6Kn;IaFPx6|_2Y@~nG*QF1os*`Kl?urv?{Yn9tbt?V;R zZEM@}!*KlP8KLQ7vl`h6J*$_RmGaY8=U0(*y)%MEtD{PCN1l6B~+xvKscXbF4 z&{icn6gE2E3JlS%km#B5p_x0xyBxt~67(~1DnN@Ksj%81o=IIHs4~@`(qtz6!BywF zRlMWw$fKt$(%WB&4i4__KFejAcg|o2%87u7ZFe*teye3p5a?iMt8egwG!|=U=xmV` zDXzFvhJx~E!1DudTP3l)M?Tk0cj=( z65r@ci1&WE_?jm|6h@P-Ae)tcX?2>#tZ_aqtuET#`P{eq^#N(f>jBH#?YV5IRy<)5 z-q>*0uAs_3NJ+6iCfylT(51Dt$NHhe->WMM<3lI>Uw>C1&jSRa(QNOkU=)5JV&9bV z&hasu?NOoUM~~%H222fj2Sb@(Ijpb1`YXFrrVq6PeJQcaf(Nc{J<9S`3=|$8#s|LJ zk4jx1WZ%zwmX{~(sF)S_(V%%mJNqwWbmq-YRkn!sdic>4n$Y-mXc-#K=$h8;uzdP2 zRP%b%{_{(3Zpwuj1dsWw$Tn#-*}@kFuvqMXbG|H4MQTZlR@2~H^uuG|j2cc#%u`qn z1M2HI+UjGa@Xg64)|Sa;(Yz|fB~Mup1EPdxHohx<=g~~4GH7nzx!aGk?ES{JVDLOa zp|JdpIG2BNn0_$spEXm%k)wLXUMykGu=qBCp(81!99XGInT_S?;8IgG{o~|kiV{E5 z1yux4xI2aQCOc~@9iw%>ez3d3s9d?>K|GJPTl-d8*(Pror?x!OcSn@>ddJo%+WIAT zCwiC-K+|ar2GrYrf@GC;SOK|0`YWAT4``44qOjoqOe z6O-arcao|V3Sa&YDy^8oN3YTU_3XZgcO(3Dc z#D>{vaXP~J`o3TRsRME*UHvCt0Fs@gRjk`^HLrK?pHvgd2FlRjFvsqWT(tReMM!lU zS&pK%U{0`>(1HmJ|8%;BzyRV#bR~;7m-%Q4xJ3^O$9nr0AdvI~EXKS9^}-hNyu(5j z>fMkxppR7CY%&OeXXX<;XD{;YPt%2w~)=O zL%5ohr;9{Y-cQQO)^t`AY7bs^T_71}u7n*fQuN+uZPl^oc0~m^1ceiO05oS?N z(?<1?smF2(kq3dI`o9r*01!ic>*H80S2=ah4<(Sew`0;h*o$2N-1WU<1DGOwq_uKd z8R;X^f5UG;m+MCQ#o7G+M}dnvf@KG_JQaePjq!PR#s01K!>)gcT`0NLyn0(&in3G3 zjTsm4VfT6fO`5%`yDN}=JWOYcfbU~>^B`<84{oFrp}Hsk%8{A5(buS`N}9_Z0RUJu zpw5k?EXO=Fh}x*Yz`!=GT~AI{0f&2{-=0PHu6V@6ql!N(A+l+=RULikit5oh)D3QU z5*ggyf{t(_9%#XW_?eJbTTQ5Ye2%o`lJZQje=j&&nj6NJX_gkgXSW(`h!PlX4}im; zWodV0e{d!U$5YZ}a|Rf+kTNza)Ti7r0ztTJjl)gHW)hMc_O5}#F_2~C@lgblLqNHu zm2b1jp)7d`MqS~mpKx%yQNI&b_NZ)M-f<2Q+dcZrd*h5bu6c50)A%)XF-O|dQ(_oC zSKaV*Eg>6xd8&c_{xWJzPPgU-yRqJX6=S4&aw{AntVTzqxSJi2(?*Mn%Q^aumP75J zNg)}!ENB=$^ND+ll_Ao7-+N~8Hgy&(nz{Md@Bmc0s}84|K4+P_)p>TDX2){6L65N( z;DyOVK)@8}bKm3h$LNUv;Gj%wc#O}C3jtZWCn4o_=bO+b7|T@oLf*$ybmo|bh$>|& zT3MkU7!d32g;$)@LnA+K2P>xr#WMnyY9d zs^5^4dQ!X`)-HJbK!am4fESN!Kd{*Ou@ioB8wj@$-n32(os?u2m#XTQwdr+mbf{QD zGXXU#023QMI#_#0xn+ry6EZ({tVrd?LXn|rcw97>q`>G$d?23*prT~JCCOom&D8DE zeeu+WnPJGSl6UvwcJ)r$!O!kb0irmAQ>p~}NvuKq?jQh9+LCSSJTMUYSYck+@Bcxg z@?iYQ?X&lFfVeutLreH7;F3`-IitB9&PjP}0z}x)dR1x@Z`Lv?OzQHI!v5vN-0SVI zR!l-BxH_8Us_TOj4iByR*U$J+A#@D)sFU1cTa5c^CQaBrrj|8VTHRqU(SaNF=fQyi82fL& zVb{r;rB_x9#KhzaA3vPs6S!u3hR8$J#wB>L8kYwd(7(NHU?`Z)rya-=BFW^fwYBjIw z&S#OBt=yLud6@D{43_#b=KM|}!x=EKIXCyyaak{Wa?u~p7d&llt?R!oH>!X&iG^T` zT1d;b1g#|5D?Z)cH*DO)zE{G6&OYKr+L-caCuU=lnD*KG(R@kD?t-_3xST!+e_o%6e ziS-UJQTJ>mIMeI3!7dcNgrFOvj*MERsEEv45AoOpcRmQUS|ExRA}|Gc+fS<;ChE%uLC3c}PFLdcm-^t_Q99Ytod#D0F>LNcMq z?_Mt4TQr?N<+q7b*@!%s%^|&Yg%jHh<7aS;Vl50oF<1-Pct}ueAytFb@HWI^_hr(5 zyO_c3Y9df!8XuiV@%Z$1xV&u2d>ZR2=ks%vWFc!FUdufkx#gbbT+0zy3_h>Oj3jS` z>4GnM?C$s2_3iaSovpz4K8Xi9iv9|{bi^}-N5`XM>tMo4n z6U0*?s%*FbUH$6|tP?8_KsRt4QWM&H04s+2qIFicPhg=Xpf}AhI{O;V%G{&zV^pyq zX30CwD2@mY*Y;1|ofwaE;3XygqwzuR=zGXdsuo7Rb@JEPcdM%O?t8B^y3zAZu88xF zP^jq{gfdHE`vg3_7SeQhGg08#lLvhc=fbN2EtEEjd+z;aHn-MRm1M@;4y| zhl;#5dh4%h1zVp3vct06}%uF2My`da*b%a=L{rH4i;bp1yhe0Wg5R8YH1I1EB>gqu^F~VJC zIS$km@WE=qvahxAz+*-eEc1ohC%RX$2z3^=tJ)Hg4Zux0zf8NIykHVvJ9|Tc2QF3G zrLv;Fcmk79sWgJ_tQYtfU*@z;Sw1a$T>w#;9?u@z`6*{>Utuh6b$Q7JH#X9&BQm`^ z^ZUq@x3)l}wk~ku7dl*IX&R(4hr|0-Ak*j36=OIIDt;agUb(Jr#-XDAT^i;`JY69n z@aXnu#1=o5mM{HTUPxXyMCu?Ueh;vVUj0{Jb?fwau(E=7^q#S0)NK@xJ-7P%0vFr$ zn~3Tc0m87RTeExoS-gHI)r3Cq*0c8No%JrmM2&v5fSAPi99P7qtt*e66b&>xPvLiK z*m7AdFK_#z_lqH_s>g3e0~2po(uran){Hc(XQK(TA!+IyCf!&*r|9sN;RWfo+xxTi z9>!nzx53JI-II2-KQDfS5RKV`W`hGK{!hTTIuH3t$6|dA;L&V&Nv3jkSK=3EDnimI z7Y4sn>!xy`m8O=Cq@wP%czf`6ZMF2Ktq%&0c9ij*nsaw=5;Oio zm@)Cw?E#VG*gzkb{s^T~fhS%f=@5HADbg&ReodDGEPEbq#qQZW@5R{}R^)d|dabj_ zr%Ol2i_?pbwURPpMQE4p`j8sUw0F0Y~Po^6&Si%4 zb&8T&ol~#RXlnUYf2GHP@3WRI(9NZ5T&NWp6IWG;IhENJnKB0NjWH_V`j}UVk`?Gp zS~gWH0}}{A&G<1`EkUo|5W4N&7i#XjsLsgRIyjTBi#e)l`K(xGM^Hm^Za2Ljy z$5m6ib;)cj20i7Gfb++mW*1XZwCb~~GeyR@!!Lyxces|I4sZjoS#vBlKE_<4$1l>++EHOXlX!yS*D;3Dn)5jEWq0vJenO zjI_Lam4q^r+Ck%(xG^W&3GI1bVLiqI2WTsoE!o5T0eg__7f1oW(z#DT zv>u&Ir21pIp*gyDfPBGN8$(jT`;J^4V>ePR(MI zD~77pc-HVxz+JeuM-$>C)q}6DG)-oty)Oq{;e)X?&exRqSPTwd;;IPlhx9PHkahYa z&UCOB#_GMBABl_GC1njOoGlzhH2(&m0oOv=nmRkq7Ctn-W8=-`{R&KDLV9W>CDC7- zKJL~f-XzKS+oMczKruU2_g}hYS^apQI5C|jxK-jv`c_s#}8@BVm}3e`G!RB46_ zUu1CkThsE<0WEsG>9|l8+Yt#5l`)-Id#iIO75oFG?XzMFL)& z0@jwj`879nEa4dG`weQ7dTm$f&Iife0fDPV*Zq|lmxK!1Y#r3cU$CHUT*JST_`Ty%r;L}G$Yi^4XJUycG*^0O*WquAnu%5 zGObj@LiK!L`mQ85%EnkEwL{1{0+FC&`33FN@a?IosIX)}?jF-I2Rk`fJ*ZyGZIPdF zCVC(yKibq;N-k)r*H^z91MhxOjkEIc5ds|(GtsBrnHynP&Azj*-IazU-v-JDhJ3cR z*-X{~^)tM{BA%L%_b|2ESY2}a?+xwEKvX1N86Vcf7eTJ#ZH6ygyXT! zo9y}42^uPn|8t2BKwOy+mWqM`7oW!)jwnuhsetIMdYfm4fYYkox9@#Pa`-UKZqAS) zNJpA#`eJ8FFP2zW`x7`+#$#lBY%=KM`_76iS{lhAfGvd}FAZ+@ zNMnYhKhx-jOV^Mk0t!@`E4)N{Md-x?j?X29RgxZ@hgMJq=avrqHr3rIoYCiPIJ3nt zKBq++UG?nIU|q@u(73HcGpA8meMeW-cd4$Ts}1cHbh#VY!`E+!SZP%n9#3zt1#u^i z3n?Db*?G6W;y4u}=!y2P9TYquA14i>6vyKJf;35EHh4egD=y^8PrqxHV`+n;wG(3v zo3s7!B%5QT;nip`6j{9+Uak{WAM|7t#&XE(fa-OVCCIPYp`op9uuR1i*Bi%D_vC{t zqi)wVi=O+5t$b^Ht9ZnE8mg&nra@zTwM$sqq92j=8C9Dt*afa*>cg`9mv|iaw5TZj zTyvB6P}q!mnT)q9+8>I??jIF2@#747f=lD(0;Q9|%bI*a@x|cBL1oj#jVtRJ`~Hk6 zD(VArLN{rv*L5dvbH7&QC6V2PTs-=a4uXxLQ)v#f_S(0V3&qJT0`bUpH+T588tnaP`+HF7+yUvo# z;iYvuT&skn2iGhm!JMLmY_K8VEI~825!zZ>v10upeAwxNktmpXc%C?`mRO$Bx)I2c+|Y4mq&p&IM}74C_a<=t=IM~*I6E4rM~6gpoi;{ zS+sE^+Ubg{u5Cx%2ANQVwcboXI7h;WImB{BH-oLv-)00g8Gq%8RQpWul-R3c`q`4I zSNPueN#4}7=fdlz`(%i9=j>*A^V8R{^;LuSG3ZgC0%Jw@K-Trrl?Hfl^y0yo>oYEa z$C#en;rqj?`MQyT^ARp`WV`2{gGbXjX`?VJ?!m5nvRHMk5=hGSabgEoQJOz|G-L=D zs_`#DlulI%*gQU@j8qh4u1}1BNis1fl(hXet~3i7%v48?;DLNdr45=V59CHBocIP% z0EY^5v@3xL_%`$u$>2=reommn)zpqE1E55Ul-04JMs`PU{8WMWqc(!Ww7?jiKsOb5 z(vFP1`~HroJ;>`MJt)!7MhH_Aoq*f*_08K$T!>fRu~NX;Z_0I?jOn1F9MhHyGMQI( z-ji-p`Cvc1t}TN}n1;fVK7Is_GJc`fUI;U61znL^b19q7<3nhi|287*G^Qc_V@$mI z;)?q>-Dc;W1Xz*cXs?9S#AvE4KO`c*%CNvpEVAb=F$$ftH=1E-+_sAx{Yoo|i0DM& zx}?qGLDVEsx-S}w_h+-U((JNL?v2v1SNGRSvq!p5t>Z2>0*_aBMsRR?EUz_m7h;cs z{P|%{xfzLi!|(}TA@N*G{m1n!Bnl{Y^SmlHuPDtJh&FzFQdyg6C1aI_SJL_F633zu zI_b~~QwZ4pn+#tFP$r|eul*c-o_f^Y#Qe9gcNK3D&BoKvd14~Nq z@d55%WKGt6n_{P@&}tCOrK}^Av1ksKQ)i78UkHb4qnDQ57Z;wF^(k4;bONPhn>~gk zPw_#?>PH9f{5iHW09Ka9HtDXcP?Nbl-iI>uG5M9?;PO3{{;mxAsyVsXVUH?iUI|9$ ze0DP?1uhIOnOUZ>QRLo`McDVC3b!qz4{v$;Ky8PkV#FUxTo}?1k>^_34tMPpBL3NG zBnEBEKf%;cM^Y}0w~D-CzFXVVxV_#P03?A)AR@_}8RW7vfn%n^vGI$^Y|T4orm-7AtP==}EWHN2l*^l^&hID6{<3MshJv2ApmHr(2iStNVk1kHF|Z7=xAcbdPraQ6MNn(m4>nJL}tYa zs`26Dr_Tb#>-br8&ki@(kf#}Hw2Fy!i8xxFI)M82ToK^??H2w!YK;}EQ)>~t)bPxl zb+ygkK%Lh;Eg$rI+E-FT^XNE>4ju4CGlID@@v3iks@;qicYRHlDlN99&cdC2_^ww= z0p6EyIPI3@Fm<`-xmA|>2~#2@h#K-r^hEeUYCrjEV{Z>`SwQFTs00Wd-uE_XZ2Re} z&Q{GscS6a`YtoNF)UN8A#A`YL|Ax;IT#e`<9}eI92t0$+%?rpSa*Qip#JOq{f%DU% z<0g`NOR_({AwI0*CuKGIL#uwz@}DSL(odS#OzfDZs(P+P(Aj|Lh10w})=5ZjeFIGf zhh&#cwf8zxJk3Iniq$lhLZu+CHE}vsqM`x4COkuXOv-`zxLZO&h4_H{BZTytm~eP| zIaT0*mW26d)mcg>YQ(ADO;z-GVq>YKw{6SErna%yxkfA*JOm#ARTY)_02;&31gd@^ z=)7ed*K*p|YZK&1FXY{8Ql;VWGYil zu-mDco1Bt?Xxd!CD!OcJr48K>ib94;t?!){VvCJ*Q}Xi6i+qMPx4U>;{HdVmxmb!8 zQi=sFRs(&k>x>4iv%1+cs!pUz@?R?_SRsI-w93?oLk8yKAH2wHKnsyH%WF@lMoY<7 zba=e_Ooz+ErNfOl@V|pxWQe0k!N?km$)mv2PCg)89+ZAZk{s-w1AAqzjc5pPVf>2b zCnMz@JboCP29owm^u(WpTIkmB)vd*vxb?qxk1*pA0Q!QWeYah(UBgtQ9oE z7KG&th<#OcPE-inU_n=T#76;xi6l8kb41VHf2rUK@P$$22kO!8>{ zvo(`#2&vVJ;oUQiy=Z*#rtPu6d7vR(#?E=$=*RbpC*5)_dR7df_w?1ukceSjn-b<- zY;*N`wgY{j1Pd}X%>*$ALyoKoYmSDJE*TG}CbJ`fNS(9i`dZO`p7sppiioEev4x-; zzJ8Nt$*1{sM7sc|NliAWOIQhwmtzKpAXf4})=Ui{gdCoC4vLO24cI9e(O5`oIa{NH zrJ``9VJ>!zKWZv9u~keWi*TseMw0mnM%RNH8`aoOqBGea|8~{39pI2G+>aGYGZpob!fTjnq82aS_4~e}jQEMw|nIbn^h{sjh z{z90}iO*s96Fc*G!&-lE0wzEUwgwlZ!DaE^rk{;tRin&NLExwfmgUE$H}^D2^%qp0 zcz0%q$zJHtqp*1Z@}CccP*QQ>-|g3XLeSH4^Tdk9XN3-tQjx~5xfNlAdnmp4>Za%+ zf^tmYdVA_R|4N_l^j**^3W_ko86;5Ix&D96#|BId=ya0;YbkJdd#@*@o<(K|oVvaN zp~}S&_G7nftLlLSA*IxFz7pWllM2go_@1Z5yHCF=zSyI{v{gecPeQyS(9~tihIB5H+!zUZw6d z*V6S;W|bN1WJhH@)4eR=M4REj;U5XUYzc@`H4CHH+iIK396)NbwhQz{uOQ-h-92j1 zrK*$DM_650Kj=qIlhBE$CeYWqjZQ~CyaemJJ1pVGmi{9BHRb9Q4cJoXWk6gcLn9#k zU3>trzIdCv_-o$;oRY|WSGkj7^QUH6wSTtmkRd|I$G-+s5@4^K63cpBpF>JQ7f}8f{(umhV8K7(iq&)+f?0k;mhSCPzehP+QH5#_VfyS+9uwuuy zQ9^8SJV@*`AU;hho5UA4NQe@LjHJ)aZRfWA#}D9!D5X8nX#}d@4eV7=@pBRoyKz{m~7lE zh>~FbRA}N9?#RF07jzQ$tgXa$FuDd7UiU~CWPz)T<)042*7X&^4)4C<31fsD8_jI4#dk0uO@VL?H#6yE>W zh=5#jo%e!iXrJ8694Cc8K4_+II7?%3R?Afg898U(d8Vg@K7Ek;d+%7eN?d;!Uy>Whs%hUd{r-Pj*H zC|Ya4^wu&!2k9?ppNN`l{t*d)iihTN$8$FbzLViU&R;y}&)rVJ<@QtLuv6g8i03oY zP;`dt?9W?Gz{%pGE8#~#ICb<)XD-6=8qZyZpC%ft*t*mYw*(2MQ8AyL?cMI0&{v^3 z8Q9B0bAiTA2w!Qiq|{8sHFMoW%YOr+NNHZo+gjW4baoFe}XfkH^e89{U2)W-GU zb-U~1HB}wGR&-6S472pudrKjfk!ZPu)$dwM`vA1~kmGtdb_(iB67bPp*iMs1_SJ2c z-8mT}#a-W5KU`jh}}0LG{;SvDNM!56%JD?8#D{#@x$OkB*%H+~*1!R&6mkH?Fyg5m71`RD z#uxy62L5O00&_EerztYrF`j7Q5oXrDqq-t2k6;SBXzuZl3V{#blFj~&|-ql3BtY@3{iW|k*n2h=s;I_iU`BCTM;v58l}LMz6i z$%uiQ+|@JjS}cLDe`*1quN*Im&=|UFap+5v55w>;-DDflk#WNR{>JS2)gG)R$ZN7W zLxP@1e^g_|!Axb9hBMed^53)Pn~)Y~=>E4n8pLNx181w7mk5oiw|h?yAR#eW1-kLE|JD!a=@&$S{FXGp**gg94kXvk<8hCfpUN0uf> zd=?5%Wr-pvQ8clHa8qy#y7?gfefQbn>}vcKxTWwvxfSPJUaC*1$#fT)l|kXl(_6=d~S@ zN)q?Kn}`fyg3YxSU+sueG_?xpUDAunatpD;sm9g@@^t~VH`I)}vf?DYjz$ww@$pC- zq|uHut0OK$^WljkwBC_U)AvR9+L?4|v?SrrDkoMr@3!4A69L=GEX7)wQDNf@%V8+}Yz7weA{Xpm8BZxoxg2+mk$*+%c4*XVj* z-=8JC#eLDp{(;`_!bTRjRHBVMQ{dH%l1f`0d-A8Yu!10EpB7x~89{FY-JHM?Q8zgD zP2{n13Q&E6t;v_slh(sFVJR=hcnr?%HJr~9UsMVDags#I0zaw8>y)WBn*{Mq<6!uXf&zM zBH^`paq#fE6RH+d)6%-TCQ<9I>r65>OBPpjp5Cs_cj7VfA2U}o&$>|j#U#a7;4<3Q zZ!)-Hxi^o!!*sHuFF2SBauJp1sRBr=V7}F|3nv@0V6!@r+N2*YP{E|u{zD}(MIoJR_To&J0>A$Yq1v2vIB98`{^53Fm zKn&rl8^{vU^SZonoX!ZXcZf|PS`+I;{s9I^H0mo zO9kp_;e7!gxcucj*~SS9x|Kd8xQ)pVP!8t{%TmNvmc`6YT( zIiKv`=qF%CLNF+Qrhd$%=Qlx$&=5j=JjNqFo8r^F_?nn`)X0@Zh~KsPDM4{Y0w&pT z4P+)?H*?apljKQRsqrbR^q+hJ{^Zjc0sYjk?cf@7h(mFXwKYo^?C%8JN#yc^i@RD6 zCqkO66TTyyOu_muFtXm8u8=qTlk)`>8-EThjRZ8bv>%DAD8IoM#`xRs5HFtI670Th zVuCvyQ8FM0?<(H-j+;ai7k;hZR*EAWdearHu=MIBET+_OmsR{vy6_fYIW!a?My_!CqOJ&4tX5rW@h3DgpHbKh@asECP8bG(*vu-H^Z=_yly!)rk)f2 zI#A#@lg6uH!p6T>U57@A|280RL_6k}F=qvwuxtnjDxT_H2Et@QS}VC_N%~q}3zyZK zaVL10N%7M^>P&b&Dn@x&d|jST3i-p!CWgF}5e%YyI!~DS{WLom4I6lZ;Bf8QUF9J_ za^cK+$ez!PR$0`0TuX|YS``&xfVvq{>9!K67avkLS2@)2A83L|j(;l;WD9dhPaQ6p z%+Ca;3tswUXB{*trkGt_1K5<8A17I15)!NwCCf>@oZN%mdNvDAf)on;lUM*a!GEmJ zAi-$J&4?y(_Nrgz0d(3JEBHw)qWfxm@`UZ8Ao{}`iHZu0>%_;&M@V-8lZj{Q^D42P zlxO$TYIRW)tLG(BvR@17w>}fhfC&>*$zUE7|BIb!u(BS{c93GB#!Fn0aX$B?vY>>! zor&;_94d87(TxWztZeWnRCMjMPX3BlA) zf2Sh?IE6aGM578Po{+~G(CPRe38~x&MGYM>Z3ddtt0&I{XhJADTums(iR){y9qPp# zJ}}Z2r$@F^MTwrW(R*8he5{`i<}U=El%V#c%m8Vpzi5VojZInG+zP?dZ$f&QvFV-3 znkC-2gu*t*>KK}ng_SRLP%AYWz^yVGnnKU%>3f4HL@@4k4s(gk2c~w-S!(piD)~3) zpAZ8geroCgk+xqhZIB=@rGce!Fn@d~8Q?41^Q{$&yxq!&MVTMWOp_f^5W#qnF7eC0 z1oX@pzzTb|Dr~1(2}xP?s1zXITam}BnNapWiH4p zmxIl)p3R*L!pE3QAtf#T5e~(`_=YJCZ|J0bNXiZ61b|j2S;?z=JpqWd_g-CH^HpmI zGIAigpW9o#?xGLqH^3ghjiPy_-+rX7vxc6H27V05wVI#AL=h7dQdDdV!+`GzS{Gh7JY};ln^| zv41eUvHBC;>IEh?H-J$8vtfCzd##=!oqxVor7{}`1$qp4n}rlw8y&r(UQK^ ziTm5rwaw$hBX4D01AZ`jcaawL6>dn{#^vg~315nqrM2_q)24FH$^4-SK6fQD)5~cZ zaQAi%n&=YmFX;Xq#uUUBo+XS0@g#!&U;z>g%uddXX1-S!4W1-W^6lJsSSy~qxH9!Q zjpx2^k|qiECvhunG?(mrYE<5zsA;J*OxTHgcmrN%Vn^R?NUU!W`O%rh@YQN-rgQy0 zfLRdZT}Sg=zx*6PY(d9mT;r`0T)J8rWB)-rKOWY%7q*%5m6%%V`Y)3|nq+i~w8MPLR{ALb_Vt!0y_N%jApv6z<<1Y{! zQ)p-VOuncqcC{PPD+T#$e%x4G$#$Ynck(tRWmmYN$&+sil&uJ!0Su+hM2xk~OBVSl zMF8chGTB22?PhJl`@nev0 z_2enj)rsPIx%Je!o9!)J(Fyk&cFR#&iIRp(tLi99AT5|gOy5fgmM+>*w!c=M@heg+ z7Uu|-c|eTmJ*>QQ-^OF=b=K0-AS^tbFM`MpM__XXcVJC&(Q zpRG+>la(7(LH2*Yb7GE*A9YH>ha|>l>qiQ&3;XMvnR64>$bV%G4m4G}!A^63G1Ds~ zbbVJn>&RAfH({yrc(U>0EK5fHOIsE0=z?|S-O98@IoFp(DY>3V%Q6+X%&Ic^ACApO zI##pVaLHY{H4nQ`GDTymbfbE-hzLEa0TYGMvtKwlQA4-0qJps$dPb%~gJqrsApc>r z!2%Sd+=9Ga=ui9=cu-a2md|$^tqi;`+A_nHw==yO-WS4)l6)C6%jFVoM|$ka`UQO% zo;|H*=sZi*nAJbzwc}LQx{<%eekr7n+Hbw~jJAJzdMVSiK&Ro6MvJf7(F?m=!df3RkLd{z`Q&*c$%pcLP9S04 zstS%@D&oHU{{BJU;AKR#32#9({r=nDvi!cK=2ERpB#`gQO{=X1y?#YI{3`m&!ag6$ zI<08wy06uu%vRhy2W833vpmC$D9^tz)=c>7V{Ryc6~??k%=&xg{5vE-Q^)_YH|;0q z1P#2PLR7|bKs>&#d+@T$6STbQ^U_`z)dsqGP#iC7MPD^v%`MJ_ndRV=s?lj>xWZ-d zSh`Nm;O8Gc^y)Pvn2F?Vxoa0UsV2-^MC)~ zulCR0V?(3;8{XeM2k8uu28??3$E!bw2wH|ZkdShX%(TM)g#-F0u?6ev-|zg1z!3$R zeXFYw=Fexp-tx~L#QqoC-}nc_1jiHLb4dTC@NbZTl*a#qQji!6Tu6m6D&nv8`^T?9 z#QzV<|4;n?dz{~j%5G~yU%K27`dfM`s-fmbM=dxx-L9nop1%d$pdaosFyGaK^>}81 z_Zrko{kQCNBnf_2jNTttMISPruwbwFThQv>WmhycSxG diff --git a/docs/resources/fl_diagram.png b/docs/resources/fl_diagram.png new file mode 100644 index 0000000000000000000000000000000000000000..cb5442732f2bf5d17e77485223991f08ddb432e5 GIT binary patch literal 188729 zcmeEuc|4Ts8@DCV$WoHBm!im4A^X-L$x@7c7h(*u8;0y{sDvUROU#TJ>x^BNsAM;G zhLB~9CB`yic^{qAIp_C#&-uLnzJL7k$vpEs&;302ecji6E#K>Vdt|7u%>v>E(a_Ma z=-jw|hlYkBiiU=sf$1=CrkU#p0}TzWjjN`np^m1eprN;?v#Yxk4b6q;e$U!;>FiH@ zOmI4N{vLbxyJ&W^@R(QZeP7O zGr?{5cq;SnDr6^F_)dmyZt`{1T1j+$H;(!=LVMzIjIj25CFUo({H+}BbDAPCUD^3n zOuBcTo(vi2&mZk>uOM5NZX6!t!6Z_-s(-MuT21ZM?79_b6})E%&&J&Jb)E}+uDhG~7Wz#i^tavNhT-wo62jlj zsITcCo&m8PN8SI#v?kYXy7ykhnD{93P`KYcGp&>Ir{1vVoqeXAsuX$F_-Ugy>)8)S z&2fA1rWkBGt#D0LdyH%~=nV@!dN$@ed}{whMb*j2c{^79CdI_p#dhh{ZECI0UF*Km z#~;gtd8RHq^`iLcNo*gSW;kApIi9dsaM}h0?%-smgn>_Bj$XvjFan>)f&Ye?bib}LL}k+dI;IC^kA_CwNK;1#_%w3#c5?FY zx##Ii-TL&3hK5en)!59}Oz)P0qbF3t{;sEklSClY>!1mZQlJ8G2zBzc7Yu~Dd-x~> zDxdv%hXQbXa9Z-L;LlrpA5vJiT2crRC-2C8aJ)UcM|2+#&80GVlU_#JY_Js2~T| zz9c*}PF>ok?!^%9`d_IYdI`OPkxa06b_rb*>l|AAS~Ghk!_&&r(h?`lu0}(9=%0SQ zvk_ETNO(qn{a>!Luh1Q8+&L@wfQF9gAAU)nRTJbDV!Imq=%24Mu`{hH{!3e?=7%)l zUQd;}c!mG**-V0Jf?t>a<++0FLO{o)f2jGq`lo&!oC}w#r~j90YEOZVwSJ;+w?F$Y zodX8V`VZs%`Mm!P`ky58znT6osr*m9|7mUgZ%zM~RQ_kk{?Ww$&wBr-Q~dwgWj$k} z{aa^+ZfCxG*SN7X2EpU;vQA%4q$&6z&Wng7G+R4m{4EAKB|~S-l#m+xnD4h=!3Tva za3;yreYE0!t6$Yrydn&>GYrA76uJzct(84^z6xMH#@zPzmiIjph4o4vafl4d&%SXq zwYndj?)~Lra$=&RZi+;?>h8Qj|Jc~rtcTLx)>Ia}ez}ecyR;c8up{U@g2dDfPhp>L zH6@E%YJKx4Hux(<$ToWbn!LR;QB3@GPOU;sTRRq$$5QW+nUa>83cF-sYb#)3>f+*( zFmiY*c4Nc8nuHx$tS0PwWFQAZQP$P{<79Ne_Jgf(M*gDt-uD8~YrtmBzrYLzgZ;<+28MJcYflWXJEd4Vn_BxN>7~iG|5#gt6Pdtj`#ZC$ zxRw^Jk&64-*0A~9jjvZaw{Cvx;>(_HeC_hq6*P~j{GNJ2{wyo~uUDzr?(Z&dZgzj+luhXCt}Y46SrVZ}|GEC9 zD2EPjdwxHu_4++x z{o%en^{k~r+&Fg?A)sz8y=x{Bo>f@b-d||xLfs%KPJ$ucb)*s8rsA3LRrl-4iFlu_ z@&TJQ($#nuzEQ?MMJVWhh$+f(w)p#vV}Hmn{Mh-GZ%-sC;~$1EmOrfc_E_w2O_QQeH6cmKL`}w#L`A5_djzy?v za9Zsig?McFy;#^*lK_P*X;Jn`GMi?LLd4)cCyI=WR{B62A{a~z4JSrSE;N&reXBRb zUwv9GZslhtEiZ>|R`WkmU2WzVzoe)*Vyc8JjG>5krpmN=)NMP9iHQ-5l~n&0#|YBG znAU>-@>o)r=)!iU1T;sKmuuEEQW(9@oIn3Dl#H}xx!cX()`!QN>*zeHm$;KLK2{wt zLLJ{!37(AR(%5|RuOmS5|mMmFf>*71cQ@yN9uRe zF1{Hnb4}d+ky5{|UbK|&rBFkTMjH*HzRcUbs9~0<_^^*6mZ@?Nr?3H2wO4`&7l6&3 zLFVVL5QDcSV|O=vv*{KS0CDC$b~*evMGdFnYz#j%`=<>$aU_441gt6VKB1fX`b*ud zoB09P9x=a6PLAo%do&&s9I{U(;GiSe?Xj(}{pJ0k^nmfx!@2L?ovogX;aLO}tW?|* z`*oA9|1zR3?y7uxqf`1-dz(P9qYM=YZk7F=HNEhy;c^Ic=1WqYc(g1waMht+4@`BN z{{B72W6iW;|DsiYl}sIVYNum^IE)O{?%m~ zI_ZnRVoPLGWQ_Y`v2{yi+8NnTc5A_4lp==4No9}NYK;mkwtsI;b-&l=;+wsml)4ko z_2E(V6g-CFWmb4Pl+v3qETX#o`C*&r77JTsWVE-pcS!WIiz?MT4FA17ErAIh=_Hhn zEc{I>@7C+%R*U!1$tT~Z`=_?Qk?(cxd^*0$Uf4uG7~EHiG_e0r}_*zrI;7hBbG=ItB}=?@UgqSd3wxa{ey$2 zs+68FBkr)q4ByCix(+1RUP-E^>>(pi+)$j{_&uZd`lS0je_KY)SYS$vcei}j|Bw)S zF5Q4lFzJD?HM0fBQN~EhYH~^nRS!vk+SrVHEyfU>yF@iq^R<`d0HYhad0tGctgw{=!gOO!bmN+ow-OPb8{7oiMQt-JVgV zo24y$8U2^nbTu<@}mzWXC0C92vZ`G^9i%|0W8A;6p|2CsK(i`R&h zM)>QA=0DiN)$bN^o;=A9O&X2=W$6C_EckGP!w@sBxLnh_oJ`j)8M^v}e*9xchdvcp z&F<7qz)1yd47uphuC20?M_FQE%Qk?4Xh;y+P*PDD2SSJM2E#gJ^*t(K--AtcrwGGq zVpy2D+i&Svo&k7Wp^a;U(!Z(tc$);vd*x>DA7!0sji;qO! zo+^r?n-)B(LIy^5;b^~>OXpj@;f+)oh(-#v%FxtwU_ejPyOwShfk5z&yl6UD2EjY4 ztt3p?E@Tdy2~zZB;QCAC4i>;f=Lv+e^8fNnr)+4!+lMvnouN?ZiettsBP(xF0({Xi zYeaQtk}Xz3V3{Gh08xQAWRAgNu_~ryz@<=69zAfWn{}JDaEXAka)+orKv3r>gP`DZ zya6=Q|0AdAdWHB-ED(fT?TgT~p})^kubT49@6jH5X9{!!CSVzV^v`ar(j9mqO?+*d zEcC~j?`Z34%BbhqMtnd`)6JVVcP>9UDgGnzJvk_rc3jG}=Ugy>IzF2+Ugd{`?-Fej z0z2d5S*bu^;|X#m5{V>n&mi6A8U6Jy>+8PtAgEACzU1oqd@tOt@TRk#1efcDE(r;V z4Y_o`esdMI_%hysGrv|pQ}bIIDh4BrF@ab$0diI20BZX89p* zZf3>um%M!Wvdd3oEy6-&XRB3JmJBix#H`L6gbmlN#(8dux-sIs|7CW0tgZcrq?E;9U2L-j!I94+1C4N&;4ni{_{@3 z7XT`IjuWl(x*{I7PuzJ^~oA;=!z$e5(5 z@n@6&?Bm%Z>`dBsCKdjzgCGW`sOFZXCI)Rnvry8D802WAAsk1KNtUnYTxp342V=m`D;?ne&^UgL> z!<vf1))@TQumHIfL_ z8pU+5dhDhTlI=IG5z6YoT8OzW7_Jt0JeZTgjR3c{~dw(`?3pHXBTAWKV)q3Z|0mF=Z@I~}5SNPXt14rS{FFd<;Jm4|>Pumk?nOPdDd5^1VVIWo_px^VTb-eGH zGsKrd==e^HIgIWn1@gm@3cCLg>#3UuD$?g{`RlFU4xQK$U15nBM^`neudFV*PgD2n zS`pt$4ax_s^99yGPuU zC#6wQhjeeK4nSsi<8K*rO4O#YKiK4lhkY1wc=lG{5CMl4IG^FJkz+ zSRd*DeS8%%D1Yxj&+amnm6Z+d?NSQ?Hw=W>sN&-Lp8{^FdwF=ERXIW)0V(fY}+RnkjI-HV+f ztXCslHpxLErk$|pon#)Ci?e%CCAh9!bA4L3HYMBbS_#L%-&OEKH!xcX9?i#p&UfUw zOtlIW>MH;tIGjI!UWK=#qXYZq4X(hwI=r1(tg+ThsuCZ61|TTV(%<*pBnZUV); zH^|Tz$~<6CP8-di7p>V zu;~*&-}1`Y2K^@qx@FB@F7~WA z4=Q%;xmf1JRV7|h?Rz1402{ z@)ZJY+aXxw!g(>Y<^0<G%)qK`@k_|Td0M5Z0V>&+51J0_7|Yh9kL#y_aFmkGQMdNdcMs4&Yc%* zkzA5Ci}bdIlG4(}+D{&>K<6$yt{++FJgjZRK1#hUZy|-cIk|<3HU@3ROt~Cl_kx~L zKow>WJw^_g>5M5rmOZc+HpgyS&T`FwKq9h+yoGv||o}8ibarDajfzP^PCGCs?r6jz8dC-)s*o_SPd3k_J zu!n4#{ArD^7XSh9OO+ztlYqrxhtjFHJxoqMse0y(#v}@LiEvUQ3r(TLm6vZPA4hPb z3k%7d&1K;av4lfD%6?%1&FUX-0_gMvC`P6+_(A|4x}qE#IntOidzp@@8Lbuzq3`aq zfu{F+%wMdORe`+CJ?C7R%pOZ%b$%ZoK)Y)E1o;TJ(V-=`ZKb&waew-EmAqd0VKVRW z?a)kK{-0KG?zrJIq@CQgi!0#hBD%?i1vdb?E&cx8nLux*9ElgzXq&@EGO_`w(Zm`rF;K4W>Z><49 z8~BS>>oGp+?VPHU$Mi5ZBUABi$mh@SS1v3S(*RJKnlbY|I(ddF>#_9yK@(_FTrjH4 zKe61uKkcKRCDf;YcyaWutB!n~Q&ck44RgQEBq*hM%UIVpOFH}g0~Of%84N$;rvi zWHNcrDywMtw=yi&!qG81tTa!~9afrWC+SA7zl2x-Van^edUk(SI7GOM77pPh^D?Z-~^^bwQ4$6Y1S* zwYqUH*DbD#)OK`rQdV9+cx8L$V2Z+@ivkAsj=<>+X29S+JQkkmEw^J8Yjfw01Ezkj zYMBd#u&dqfiSh=lSm+&j@oc$T1qQG}LmC!vTbAW+Ne6>;Rwy8N`TJw_USIy`+@6|S zy?Lr(BiQOQn~=Vs()zk$$}{8iHmcMD6Yym(&2)n!{d}>NZyQddMJt9(51ZmnjKm{mtqg10?i?>c0mQ|vczQv|S+G$0?ZM|qjGN&OzVoYDcTy3l;Lo$Y~SPH{e5*U|}JS?g5ww8VLk z&lDlxksv-W{U&Qzg;IdXm}1@^@V#qzD)sUdrVJJ zdsq6+Q~Eh?@w1KExHR_1Kpx*Mo{41#c_}JbTF+T}f`ZukJd10=5G!^ochXo~@{5-q zm7k5Q=F&*Qk0IqI1H$jJYlq0k{W;2Hp{0*otkH*LJH91;y~E*ylJbdf7YHs;%y88? zQqavZ*xEf$j&*PwcOk4T*(b610aGkoIt>T$3r#-@RZl<)!6$M^b$q=@O5XH zyn_XH$7_QN%gXw2!Jocb1e1iMCwQeWKgJE^?Iu`Qnnr%CR5QXi_Ih{tlg_qO ze4aTjeo?ypN00E(t994pG?Nph_C+?;`4m$OqfO|xlVz!M99s-IRY+BpV7jCKN~$5Y z8i4Rr*!zjmc8!*`$z8-$dPde)Hd6;B-mq(h+pH%gtmf)-a|W)!tbf^(Ye1M7Ym@;! za3i-bWXfs5rgGE*I^Qbq=_N@JPwB~#24ZG?|3K(%GrltiPA6w_s{8> zsjI8sQ}&tdAlr5tXVeK!y?85uzLG@ z+Afg^{-OO33;o|xy$-t1Ndt+H1CJs$G5Vm8CQVx~+cE&D5%O-@j^!OgX`aE1`xfkD zEL3h~sEipi@E&DouF^}A1$oQ(+D5l*5ap4Gze@6fo=LUx^L*@-`xI9U=xp;F zqn)6e4tWr^+sk6Ho5uBSx#h&Key6cfTZg#l5^x1Uy3jjI^Gl)1)P#h_*yWtX5*yEa z;=I%7Z)O(;hBX4oDC4R+E7D=VjGcs<*2%b zoHa2_O;2niLN@frXOZ35iG#XxI(%cOZg?waQAz9od0DS`V2($mt_$@M(F1TVQsi&bM`~Ta$%a%b8$&y_-muoCPTQ$>YZ%X2=gFXG(^z z-Gf_5Fx@B1=oe}e@o5U+O}7l=N)DB>r%TyVW?9AkS#kjlrt*Ga;1k5z#zu`WbI%oD zfppk1WUH!Dsfk4XX#}@~Sn>I)zv&e_#^DAr>WNuAOj801&&#daT ze%!0PO3zEK*09|l2UY)p>ugFZ57-hVtLDz`r^XB%e;dF5GipCt3;-hRk)`s$1PBf@ zL=+j@BN}a@8yoy!**TBLGtWnk-(paXiIlr;A$v;Q4k>6lSqwWa#>?9Op2l?2)9?(c z%e*arH*yL2%seIXi=8!>yU1WnN2HoLeD?%Dq?Q|&n7_*$ahFstnjT4_MsE=w9y7dv zGV0M`_15}ITURXfTod&io3hA8CuAw;tri@ZVbD+DfCu(gZErM*#e3}=9^eJ+SMQ#H;6==VD2=i^|qtyc0i`1)PpWaN4XiQhxG?=?ih44JQ zKCqBwNr1T)mZgIwhdRD+p^9t2yx{18XH2wUK|C4Co#11V>)=LEyd!*$93C;Q%^%dv z50uFFC)Yl5*}5aX;UUcM{KCif<9{DkC;vF$ic>-iR1 zcxeRs0KI3s?^i1dHI^>*BbulldozAv9u_B<1E#44a1BDJ!!CuaV%Sdt`ID1$)BAeV zJs0{)fAT+UXZuLQbrf2+7hKV`phV7%1zD)}NUwh%s|Gdj`)ils(=Kl&2|2Djsq9+B z-cYxpi9p}5^ix7cYLhpm-W52!LQG8Dp5YDW;J7ywlIB-Q<1eTgW)P2yYgPl2U;$$i zp~o*nMt|=a@radk5?a;kw<%UdGhDC2l!bNJ#0j~fk?TMfGYObxe~y-RKaNJuiXypo zWdS}nT;Dw6QQ9!Fu)?{VgoX{3Cl5}h+`ci|KM(sMGHZ5w$@L%wu@q2tcfzn0W(0ma zLYOsO^V}F1b>$o!LhT7BQs6#cJ!-zY=OjdpZwcU4QipSkuTRdyK zd*Efa*gFtCfvHr5yha=Ftou;O$IVTn-x8JtjL$;g3&i)Lyy8ehGklXpMuat)!Q=u; z4gwR(M>w(+ip*)E#=|GW%WWU*vs9}&G<@v8z1nw-dMv6@L7<(p3G}jW;Oe!vLacq( z$1>oEkLMH0J06P&jeYe4G6dmF5cP(=R{^upn0B13^xh|&HeyB3mNHvZess~nHmM;N zJM%UL2A*q?aWC$G+7`QvC=?b?JxHTLkG&A@d|bjc{IceXRVSf0q|+9>Qzasa!zru2 zawpv@iX@L%r9GTMzpLoG{Ne``HZCtAXye#UDo;+4ZQo=u@w<;=t4%f(V`8XF-O|Ib zofuy5d#v$Ib$`2aBi_TnS08HOCfB6UZCi=CnvX)EsFOlymo%(bebYw(_(M4 zMr192hfF`jmcWE^jilCvb!pLvFt9RU*ORWo*77C-0FquE=-Q4-1Dss0%kbIf``6W^ zYZPjl<5OEv?g3qE|iVo>fbNBf2v{}_rGx%Xbiv$%1oX&>gJu;3nIRaNesv8rYH z&^0-6LD*35Vp-X=)xhPm=Z%l?qUx(hqSGW+#;j`_vB|1VWOQwF#n(z82kEsXjNT?L zr8%y(+BnEx-ER}UI@`4`k=3I6HW#G(gMp#{j)lyln z{+hs5M$U%XvGuCo&S3OmaajXW3a)#!V=l8D=^KNN3EQ$90DWS8_Yqm)x_w!HAzQAD zz4-5(yzs2!K$tG-ij6#|14`M^))uEq*+;M)#3(mS#n1E!nC)&G7gUe5|0v9Iip*+* zP3404YA1Vm%wgi0n-(_4aNU@v!={4lp=v0_Nv-G(e4) zp9T_k`10*H8@9Vz93}dAYXJI^q6wcMSZuum54jB2L_D#YQ|LAw>&NjH4OA;Y9mhl! zKMrd=!3JD4hwdhKb!t~XTcm8|MTwV5OLUj4jh%Cle-a|NSVwetjh<>I#M9Nr>kEJ!_(b<8M4&k}yXNIwq z_uyNL{W9%&-^Gj9r)z{npEUVj5tcwGm|xf}?oAL4>T~S6u_21ro0+RGktSr#jG$fdphtBy< zuF(#x1$@gw?!doWX6MES6L|yz`S-d?TR~%N!zozS&C!91R$2a#X4a}?4j|V=A0N~+ zPJTH{VP7tzk4tk8`F>GWe%4n&YSTvh(jH3CgyVC8brd29Af#FmzNy<6iTcZo@9%An zSdRZVxv%i;`qOppyQ|+m{$zB7S?Yir*;KYNt>I*Roj@_RaA6XRG#N5?NuQy=EBs3F z*!`s^KQ-GBs8@vFEav?QaAks`VjG5^X;?8-SJ{YwnTT;obOb^61GJ5!-c`Olv1|Ii zx*%$(9*8}6U}X|uHb&0onj!AKSnokoe*U8nmxvFW_Z0hmDw{ZZ=R-jbk;jZrGOBK> zn0E1~#LKL+)O&ip7QFUsf;T)-gk2!T&h-)6beD$~TJ*gy=MqH!ZI=*xKh%<~>KSf1 zM7E+@RaLICH6i(1BOxkK?#SyeeuPtxmS$fz-{Sg)NH$Y`$`%*jVFGbSmKXYjG;S|Ssq(6;h4$66yCx!MX>uzL#3_! zffKZZjhVrOkig~ppU15i#=Fjhgf7ES!w_b*&EXxcJkZLgl)2RUROoEe-?2dMK zPu4r5Wv``HECj~ZabqxOq{f{|vocojUK-zCk;n*{2=yxD>(g(r^0cw_A-;tepLsKU z2X?5bujUG)@fnn2e_2&OOFB!IUs=@(Hn+B;{Qjp6_ya%9G>B#v)X%?KYRgOC-3iTd z{kNe=^cOiox;K1zqAcf)+)~;?#LRV~H#~L|k1pp+lC*eD{O!$ii+e^sDTC9aAJNg* zBf+xEWqI-50}2jxL=mOV=#NvH)Z=Yj>lW`SDhBs0`-o9S2Fmy_9@nudzr0?Qd1*+P ztsW@ga_q3yg?piEX@o@)R4?{>XT~y)yjfqZMgTH0QF}eN_XFZJ&hoD*r~@EQP4Mp& zi&MpPu*4i1v#cVE>lbMH{T9Sf{>K$eb(zGoi9tp(4gz1!j1}LQTv;u1KlL>o09F^K z8t$1&zoL!>k5Lq``V`zX?pV=7<@=Q29zqwR2}O}ZWu1j#SMN?&j;C-`!gyN@!M`Sj z)5F!)z;`UwPM6qCad4dukR3@z%%W7_FNlR{o?{X(q08>~3%Ri7Zb-=KV<&t3UA|&{ z-q&`n9{qa3bu%nY#)Iv?zU#>B0FOY%#EAS{41iXdj2De{DDMotx2@cVAz0baxzT zl3*X7e9LU)?%WXM;axe8KCcD4XB#nLsqUM?oKaoarfgcpWu#sdy@i-2Q=1WxQs=y^@c%!Mitfw?=J5Im~=o6es_85hDF9+`p4?e{Z$Kc z(f$yRT_n~;`;lLcp3fPp2t4lP$aHd{tSow#k?s+i(bH{VuxmHoKg(&y&slMaRa~}T z@9ZY*Gp6ue%?CH14u8&1PA2%W5$B4`Yf6Fre8bFE)e$XllU^%l89gcrVCPp`2&mdu2yG;H2ke7!B;SGaBh#oWP|B-PYqLelVYwwNkhE2aP73p+$aLn-WI)X?DsNii zKO*$ezQjUL`Fqgxxa4occc*jpHWG>FMe?exLAp(v+R_xseCw$1{9TN4Zd1?F(je6V z%UFQx+P=TnnI_j+AA0Jk5SaMz^cCkV9Zx*J2Dp*Ju7m#s%9n9n0w726D_45!`%gcG z|1m{Ohk!ZyMTQo#P~-H1U6Bdh2~-4F%e&Vl)!9*e15VNP3a&3sMYpI29}$IWt!pYS zD!zYqf$46rf!rznliI`Gf@=LxIQgW?x{RL&;YO%cfCC+6B=2z(2Vd%-sEO8{ZoU4h z?I+5t0d8p!wk#uo?B?i?j|xiNty~g`@~DOOJ+XTb>*_}(@ynd@}dzyKbWHZL(0lPE*9*yLjTkCdNIxE zrNZnH4)1}g;bu?ofN(II_ol7TQ-)v{CYRGb`Tq4EZ#>yVy-GWq2^ zvLAsp%J<%$Nf{vz*b=h$E0&O$hx?OkVMPE=Olfm$jW2(F#U-^Lqw-c$KJ;M6L;Zc+ zn131^;5=fvY(u&?kAV#w^r8x~_C7AmA~8u~HFwlZ@(1opRD^=kD26`K-B6CR@g zWKai~``!kwep~d*gl6St&8)KaHDYL0r(=RLzyVGg0&+KE>4b*!BVMvCkl= zLMwtO^v29L((fhj()o}wco2H~f!>9%uQ-!aY4UC5DW81KAcv!)OuMUQi_x+Kj*7wP z!fJ#~i85J{^rQJu+8k_;^{#t-TM_!@XW6krRYku#qRbn2lC`);MCDD>?dsOYkbX0t zR$a85S81;BUGtke*NFbnU*phhKawL2C}Z>9P(ceT&%3(f-qhYD%RBj=_g_FxKc7ZC zf~4v~O=rYBwnb1^(0Bpun)Bu^!D*KM?-fa2)x$FyljrLkwz}@K-d)yiy-eB6spq47 z2wdnZ*zZ(x%ZLN3*KmRawzMN(fuu4P(O$hj9i6^$^? z%BHcf?d&5rKx>zqfWo{~ARR?+r#J!rH58b6O5BA(sE3EFURln{IPN}YNx6cz@3^depYI=TuHUEDk93A^T-nDN2!zbjqwT`* zG2bR*1+YVvoPzye$73jIEqnJ4?}wqN-$ibh&1pGM>|uxr!TF%uGk+ z!C>VE7dE)tM9r6|6nz07G^caNy1IwS&36!D*CO4xiQH?eYA0c?ink53!-HStmo{_S zOm~v;o@ENAGo!=S#7H7N*%E~)9+X`U8};rHuHbJr8l4U@8a?ETq1v>Fr#a8c68g>Hq`jJP#W+-@fy}%zEXd=bC_Qet##ubP^0l$Z?v=XEr;z>H)@yRotEpH^#G_a9>%cCL)eO` zT}xkH7Y`^~l7!FeZWMaL+bm~hXE$*m+v0qwpXBO-2y-yrKy|llw^wyHQF(hhu?}R` zP!pp86VJY>OCqT5uc=b&--8waj{PQujY=6CWGCbsUK=8>c!*UJ`ndBH=fvK~cY?S3 z2dmK6g4?4}znS-vbj zaQ2?_WN^I_qG|(X_ME#}jEZtkdsY3Z$8s}G#(hE0sz$7~MkXVzqRXn0HpRPo#L;s;L9()rb?rRC1y1K?wyAdkwuHU!Ys@+Q1j#a; z==W5z91c%~L+>Grv<6@fVNJrO)>bpaw{iw^Mz2^E!LyjtT`EjCT+!TtkAqe8%c8c< zi9xJpg>y%s4DDCM@eJGDSB>GwdPX<(+(aAtxf~rq&f|;%=UDosi83)N7rJ#)IBAGh z)OT#C^~nn1O2^;nmr{d3d=e@CmGYBscosCr@@zb?!0lF6)Ocn&KoK!=a>`|UOl9^J zLYECBJzv87RtAl}2@03dNf(V+C$~?qSYYrgz&6FBFCM78x*bdpm&&=>zp}yNw!d<) zGf-tci)|Z3$h>KiRY2GC#Ok*~L~LXhCOC`)u`NB=%4*6b#%?lGh@io{m=c*|8fPU# zCYd+V2s+3d{fMKgLpgd3e3S!Qsb3tTtIP|c`xhDqp01jjS^Ss; z8H0CB2CAg0wgQrSD!a))p#GcszgVNWY z#v5+maLF4A0*3-x{k9O}qkTHExRv8vi^7!0&g3g-&jK%M1;Nc)b;_BG>3yz&4>c4K z*ZFpKqqIvP4CY;j(eQA`l)+grRn%na-bd|;r$5Bj6*Tu{*W2mRuNI`CnL|yImh9Nd z%X&7}xvvXE5oI1V+SLm8)XkCE{{Ze9PM+FvscBH5>reOzaJVt7x9EnW=y+-`8YRVx zIvTsm4r+Qbk)3aGssmVt+6~XI2dLXsx-?*4j6JGxYy}_^RyE#Ys7`g)u_c?BaUXg& zndMkiGkh-phw7X(n*H(}_JXke1!;7>Dkzt{>2iVIp5$q`%DY(I0d7YcEoT8yFro}Y z&sQ7FKn%U&{BOAwFKEQAuUwbsN zrv2K;$*1*w+@sSTm?R0@#*(v`7r5ez)o=H`9kddf41ANSf|mByS|oh3NVEcr-zz>( zC{$Ui!nr0E_7ufzN?PJU#CT#^fxOGe8)-o4kW#=_j)vikg?_%|G!?vaT;&JwzJ#~9 z_~l*FQm`!Ap%PCP9G^j?W!%TN8d$u`trQG=gE_pxT)V1T9(%(SUyO`j^4*P9v<`lJ zkB5T8uw||<>xe{SfMx#7M3wC$gr;`Ga0|KHRqp~d8tEqwtEL~Y{(uQPiUlA*JLb~R zpAeu5!-*CISKc_&#l_oFSNb$P>0}yS!k)R6oU1(p*n$sTO!TO@2^=%wdb@Ypy*xB4%GlB$3|tFuT|hDTrhqhN(@i<|vPiwO!pia%P>J&p0CLDSftO`* z;E{%wfx;}Gdc99?J%2;LFatL}7rv*f+fjVhlw92jrd{?SonIJXts=j1c3bl*vPa7J zCRI>#&FqJPe5FmC5MAd+%FKG(!Uz;z{f~=X%2I4^$*qRYT>1fSsj4mPEqCx%_W}3v zkJ#ddzVMbCkCH$tyX}w`~5# zr2s`?%2+hzX9yJy2ol=^{kbX&F}X7-u6Si(+6N{Lnei$HLM^X-8NX zBq;1RIgtw@sL%Ka__g6G{SDG{Vay#n#F3>Fhahu<0DEB~cmvTMjmTwGSeGN+Jqo|) zUzTO4%Q?UeXRwGIOC#T-242D;DY8QHoq5yZ*yEHk{;s~7Q8#UkJ(F7kaUbFmma< z%1DWI8*}-WhuC8~p@(OMX12;yg`gQ)k6EZcHz!G5qpKnE@tXEO{%Oi+Xc-P6z8jr7 zVf^*zgolXV^9j9I9h|IdF^f4rN-iIwa55q8;=gBMiaj%sX1>{(nIAHMtx$NzJSQio zVNV#{Z6T28l1^Zo zG1hCcu`AYiK@uf>XG7-AjWD?bYK+K+*u{$g6V^GJuqerxFm+7sX_r`Eo17fHggLsi z?BoDf12^5^3Bb~sbQrz-zhuE_1%(zg=`+x*8AUbDvl0VRx)attx z-~v&QxZ+N*lIKcdCud%JN#oNYG+#?|=w=$vADb8K)&at7!gy=E@BxRRbk&d$bk&I_%FRT;rqc4)wG7;m$Hh>8%23GszU1|DGOFyZI%uiCGG9O zPnN|(ib9dh51v5R!nLh$R1Zg&e3We|_kU$s@zr$SEdrO#qPGALu%%Am&rjy%3w6K75p6f!ZxwdxovZ4}DY|41);r`uS z;G6)ozLM`J`3(@~`w!i==}$YR956q{t@TKay&18fryfhkoo=j*!iX{|lOtvt*>pL? zUCqH+*0+^1$xZvQTjhRm&o5*^igx0@ZBIBoj~AkWp>i4cnhboo$JD z?A>D(>rPVvPxsGA9i?-N&Fp76fIKp8k>VAXm`KsE@8d9H;U3|oa167Bb`9`{F>o2X zi!K{e0{eqk!_9BszCEV`poRuJ^~&W@h6z71Q%#g3n?&yS9!k#GY8KJhWZn~wLZ6CZ zZXg^cwE+yVU}IIte*GR9L#@k)MXS6U(K(=3c_Qcn80D2{RPj^vGU-# z4USJUdHTNinpD)fA!fL4vtWdwoN@QieQ^>w^!PRKE+QQ$mj+4r^3V3Cc#N>A5dRNZ zZy6QU|Gk0I-67r5-HkNTCEe29-3>~2BMn1$cY`zxT~g8r5>noS-`{_&d)Ix*ykX{? zPds}+vDa10VZ*?B$DrUP!vQ0QmGfzJCVKYO5iVO}J?{<~v zt#cKGr-D&<(g&!ZKh)>Yq^mO``b+8i`{P6f?o15DM zoA#-uXo3BS?SZagzY|-#!sQd{Z_8@s;K;Zlz$bH=K44UilqBVQlKT+ z4uIJ$L}AQg5X5$z+}zyzN%{F?NK0@H0cm18k}!*}-+Dz_W`u5H{_@snqD`hTH(vT3 z2?l&_036bbe+6}W{MxIHdbjSMG{PO(KG9dP-LG%Iwir0|!`^M(_bk^Q0mofq#k_Vk zWs_2@ZEZIge#&#xJv~21k(sJ#`Ympy6by$=tMnT?mqf^GmQpeAot-P)G%_;sdDqQ! zBc7pGKl~;D8^qRue`xl`yg62@YiU`6-h+GZv0t*?8{%bC*MS@pAhrfjbKIZvFQR>n zkA59FaaPSYd5I;IE$UowM{^niW2bb4_Lwf^Q>>6L8lVey=A(f?CJ zWOMMX0AzG~so`F~CSCs6I_V#=a&}R>j>KWn;V;+FkhN@3*SRpEU*h=rGpMtA*k1lA zdw)6gG-L~L^~SDC@SsGwesM^lHBMk%`YHrVjP=JoG8b;ld^+hKrDDkqT?LZ9dpG;YtLV~qU;&~AB_mY00BkZgTH zDqsYVd#K8TR|_+5H76fBxY9}gsYo4=O{0JQBvr9hA(ndZ%P#(j+=&V5(YUy`VA}xo z_Flv4;jQS(^ShP}oWDI`eNK$`8|t}Et$hgG?qI<;dYn(UyVz(CP~(Fg>{t2b1IQ%T zxt3=%_tNwn*u%@;gdgWJA087w96JuDY8hH1C0ur1ifmhIL;iY`99?qe{8g_+2;Yns zc`V;%J^#v(`{8jee(gx)p1%_akvRuaeVUK*d~0NeZ=5b%A6h=#)yMwC!;ntlgxebiKH1|$PLE2pJiQv)g2xKrCRACyjrPC$OnlhTg< z;Y8~x$j9PBil_HV4WKqgVaE{D@^bc|b`)CC4SK%lKsM7<{jatMT%5?jI{9n*9rMi$ zMrA14pCFz+Kd%Pp#dE6x8Sp!1-F;}o@C>mf+e>JEr=1`p&HhSI2={6VlnU%%cVyZe z56&V8iWNaYqrdc42E=gD9rqf52DSV6=wlS!=5ipnusrXM)Ka1%P>v6{!FM!+om7s? z4&XRKzI<*E&j-Y+2D-K4pQ{~#Kr9}U0hqsyER}Nc!F|R^Kew;y(KJq8f5iuKHA|@Y z!n_cH-Ctsf9NPNb=Hc1iIF%;QBRbh*>o-m0*6o1Kzpb@_ryw>o5ePNoD)k2ysf%A| z&5E-Xy#=DJwFQBdE-D#4mNTU4OK;qO_;$j>snOrHy&3^De>Q+O5!1aq9dP^$98i`5 z;pnil-J{sbLUT``*%_o#cOhq0RgsEqQ1K&UG zv^02^A)5%=+g#}M#D+nAzJ1wL{gcDpMB;XiPt(5B{#|iR^SEWn*m={L#gBAP!pgp^ zpn1O4J0r(8!V$7ytB<6{rS-eUohI+42ftgFBXxoQrOUyfvqQeXo`7Fnb|%M&2BvZ= zif7HY_669*r&Gh$&81px=S?K7+#DMzowAj;8noH@OC_=9=PC%kay4*0=YM7b(f?1z zQ!pkY2Fw7G2anx~@mseI{Y@NhwKp1nd1HG7sEI`?uC>JKpb#~1r3sO;aKIqmqe<4@ zGibGNR$aq_n`kuHIwaqv5%G0xz;l8A4ObjqtQHg+`kIgA;I4*G>=!PeePQ*4MT%gX zyHaBc7QTP*)o1ocP-w~oPQlE`zV<@(6Z@<6`Er)3xm%YcY% z`Lwm9c^I}mdxK@9oneO%C=Qzna~AA0`=x52TKfQJ$_|4!dfPknTAes#?_2gfu2@HD{@=d-BtroK~j>mwoGF%S%nOWEywNr_L7a$j42_#q*ylR_D3QHEYYl zk?ncC)+|n%msX@%KGCY0*fCgQ?G$~W_Vg`M{4O+QvERE2aAe2%dXWqN$?ySO&^PwxY$k|@_Wf7N z01Cc7PQK$4atKtt_ipTd?X#(vEWcah5WbS#XdroLRG00z1L_r#42_3Mb-?I@{sjoH z^1rRqtYMJSc%_d4%bHx(!wjtD>DrawTRY9OPXoNBmK+=lDB~0BAyAC`nSIVcj}d%p zB0yuzzIVw|Q3|dDrEo{*zrI#ZZj#e@NN`URu10YRkr|UwC3)m%I7efg`4T- zknhJmGnc^vTHI$Kms4H&jqQCtJ+3=@9^3DKYPbjO?)CSfOnHy9L7uHzBDR7CeEO=r zgLg|czq-R|_oteVO{bmh-_Jk8*z;RUwb*Q5o{KI>S6*%$F~rcW6>luq#{jgy*S%&n zpbI($uWjuN?!CBY12WQEFt{4&2MivBrLJY9E$Z&$Qz`Ayqsl5b>2>l(uE4Gwqx+;L z&w1S~{NEn~yP8_Vek~O?-Uh3oEwurudX`@mr$lBBaSeYwBC-CwZ+B}WOE1_6C|-t6 ztRbrMN6O3hb!@CXg}42n&H9GbMD2j1b@czjn5fjqpB2>}4z>h|0`#t=LH8VhrE@ZD zoAvn)yPcR6A>aEjQzQr#Tvi)cndJ>gUjoGG?EeDhf%{~Z3M7?+YLK@(;h2h4eM?zw ziX)%|<#F61Mw*@BKO**h(`2km(|to@NL(V6pnc;B*zlp=ls5iXp=7^bvbJ9?Jf9pz zV7B#Vw}qN3yha7L*zvPPO5J%n$F~==TN}J1OWs?qq{KAZdGv?ZI9_R!J;Y6A>@nCc z=)0dIRobkazOs0>-C^Boo-+h-3v}oi@MT|uqLZs9nq5?!HfA+VOgRrpK^N}H1MRqf z5Rcacvh=efJx(uTJkKdc*62-Z>wf9nFmsmlZRaHljcDK=t(?M>-V`|Lt95P1=lot^ zVr?FS9nfJR-?BQWnP{8SKyCF{`tzhTp<}mjGvkass(4*+7dL_T>>Ya=o|BYdP2FtE zOie#XbCk2okjCo#^0llT^drU8g>VWPetTt(*J|K{)sneE+4^l~MViv%EaiiAY>$00 zx)c1PzMqiv8oWK#3VJU56Xsd^oy0Z|5}D5ydXEMdeyzzs>+rv1wcr(CPsPQKW){7f z{iUl#AF|tD(ZnZ7>{o-C7PfgFeX&c|apq4@vwh5Lu6Ydb^lR-atHZ^04ay9APL6^6 zL+!syPMadLyaj7Qw-wJ@k z@*jQOz&e(Hl4)9q5sRj7(Q2&ee|7uaER=SF)=GHhG-haMNo`0{(s@>IZ>pVM0?^w| zwhc6oX*|`@O*CF?+}e0BVr#Z*UkzK`G&}VK^mk4}q?aGXghN_2gpTsQqBUO~J85*@ zI&cUydymO6G+6(!d_w$z^#fh;Gsv(9Db_QkcOU8p+P1IPNXCsHw~OJU*#~}oX_vfa z;&S)h#`XAVmh6n?+2%U!pHIY=B{+|WXIX1D_VLJdmD%?lHa~Fics$xnb*(rJvZD}J z%i2y3u(P?`-wQd6HlQPYx}~TR;W~fm0;MSd<;p9+>+Q)>Ppg-rVl28bufxXAK^={@ z`=0H3yWU#!+l|@+O*J4B?54l2XSYaGN6_=)w|#Zf_FBAgPw z5O>0gqQ=h6@VXkQ&`JzN#j%#0#psJm)0sXl&gNOI6J7oEbCA!RM>}bMWPZeN&)+Jr zb*HC0ESRZmL^*js8R9lw z2N$2ITfD9Xnb}#!?>RNmf7GwKGId&{4ttgXe8Cdzg!&7N%eA5l#qCPfssp+}^}wxU zjfZf2CsrpDt9OUqvRZahptOP#p_Y!0^~{IAU6h9%x2~bXF1#_>M_GHKyLAkq+>k|iqoD;)<>$#Fc9|mn z9owTjJk$TRa`+4ZapPX&x(8h?Mf)Wv?hpL6fX-@>=?);KF?&~n@ed6_?dLEWsWEj~g0c2Rr#v!|o# zG<2c9=TA(Np+Q+~{z+{009yDEG(ktcQUgMW`OEjW_fn`x-XC5)j!5v@UfA5Y)$F(8 z*nSK$6cJj@We7Xf(?5;8@WQLgI|n%%G}qK(fWzEKYQ)*|Y_<#*LISjXE!vu|oqr*~rbO!k*mS5!|N zEI1U-$Kg~S(>b5&+Go~KOQ~KIl2SBnJEiIT=}|$q;hk8{e07%XUPp&LQ{#F zx2!Q-)1pY&f0LK@0bq-{gAOq=guENJWA3D$%KZ`VD9Tm9r>NnhNYUGOEv;{{6mYJ~ z+PwcRBox3tKRO#aRqzk%Ylu1q;ao5*qUw_VMAgN(f^$spx$Fg*H0OwGzz*8v&T1im zzG<(|a@Z_=jU~Cko|!bL!JQ zu8yl-hjy-f1;@5xVRY2r(yY`i`$;MhrOEK6>Lppi7}NT1i2JFCKtQfp`3c>o^8K9s zo7tK?3kCso&$%PN*xBkjxw=wc6kAGk4isjI~HC~$K1tzK$5YyCpAjP|X2$lkWO zQpw@Ba1<_KWxBJ@i}W(livO5Zw8JU~ug<2c=4!*_wSh)GO)6vc`i`!4%2OIHBBE@m zNPkH)k-uK_TACW3Lap0mmFsA-t*QFqgJ^q`N9g*Y$GYl)$Zt31_ty($+BWmMUly!< zgq{T62?ri?54_c^ZzTSC=5L`I^{ZXG2mjkH&{5A0KmT(}vcJwu?Hc^K{KJjw!%cl& zvimxa`e6^`<-T%5&y3SPB^vwx|B-~asAC`SsKli;I6)VgK=3fS-Baa`2ALsq8LuW% zJhxc(7KAjRZoGr~%vQVW(Ienri)c331TW7Lu>ri?4(Md($B)lI&e<~A`BY|{yyFSZ z2E+4aZ(wij*3yfc_j-%{>nTVeZVxeRa$>?SUebS4qX39gdr|)@MxySdzw5|17_Yv3 zDuOhjky|}Mufb%UANfPk`Ju<@EzTDoDY&VanDKj#{U7>;sy2&QdxwAl#b>ZZxDKa3 z_4(`C6ac4UmpaCO;Pinw_^1H7Yk%5FtoCU{rnBTxKp<4^Q5tukmREhj$S;kc~Xbo;FQ)b2A#8Mn$*OI#j3fI^1^09N`)6 z(q=7$dD_aQ>qQGjkZ{jD82fCkw5~mzc^6FV7dVk3uFtxDGgSY{Z#3gb=(BZt%C9;T zDS$fs)AmEi`6+PwH{}=-9ZQ(~Pp3hv{_WDUkJgp(KS3-+Z~zRBC`d7EtBd|yE{}ay z?K3kS9fAg;Oh@O|PQ|s122V#VC=fUsC4$QsLf(=QPkPHlwN0uVzHGP*UXNkiXilj?AE>P%Bbh7TVHZD_#O)~IibRx68n)V=g<72g^5UeqRUk=yuZ zVaOa2zKJZ7lCT1O0}C9kEIu8%ELe{xQNQXxQ7H?v#QdGva)%gEHCV4q)$G2vSWP?F zLdG|>^NGx`=?jBFNVtX$Q^EE@4TG@8Ag2h&_;Q>pY6WK+W3O*M*22#wJivnVvI|aH~QCT1pN`ayZ{TcOOIbiDj z6-|#rR(W-IcnFj@{g=?bX0~6QW!KmTvN^b(T)v2-^_EvW3>E|k2}7CAGHO`Z)fB;V z3Vja66X4B_t>rWNoPIMW_JdD;3b@R^ZJQ8Cye0Lt&cePBwFNJ?M)ZZXgY$A})0$E+ zp?*qn858|G7HY{QNoZXAS1n&P7`1QO%Az|2JKF5Bk;JtY^gj}P)oiqM#m?*{-Dohy0UF?jkG(7?WR7hXBGz7XJxovX}^hY^GpU*$| zJS?v6!nNvnv)wz{F&6?;Oo!S9*Jc0nM9 zY;Uw;#T2Q|wMP6=HZO$qv^0>ZRv;Vg;zGwrpP^%;<&36TmrziLxbi&kCHm4a`)<+K zx}ta07rD9k$g{@Rp=b>BymW`+`0h;H#jKTB znfxQ}bL~2T954Pgx?&nfNXzBLgfc(50oJ6@Csl8CG%{L7@5u5UH;^N9(K~rucQ;-} z$GwCr?Go;dp~nG5l2kd$H@6nEw_a3YGDDi`)9G z&X8e|?bi)$<)dTa>zL7zhDUsKQ#xo?s-HxIgI*IU^qQ1--Y>#+$9~mDQ7vL^HuKHB z>_6*QF6U}!!s;V$k5<3-h{}bcL_~7%5Zk(F$1nA;^5gyc7fs|4%EljM%!ZiE)I0^2 zT5uSjpb7YAbm3y=BJd{buVy1h!JwMB!aoA{8~wc~hEO~m=)6yXR$5IjGM`jCRV zBWIZ*m#&CSW+XhO)#^Pj$Dc?wmFsa*JB=)?x1TSHflm*tbW-~L5+mRb444SaV1u(% zGmF@vDw8wUI)mUkw{&v%)*NRh=!zmlt;M;pw^-kwfHzFTj3r8A|5m1u(x??yQ4~K) z&mfT8lw(?|wL;cp-@0&BBl{<|zXTMt?jR7M1Dpa^3?nOC^>$}6asRd$ek?*p20ofu z2xDHiMQ3J4cqH3<>Ssj@sRZfB;<=oZOZu6eGw4e-c~97~#(kL67xBvFGqUgZ5IvGxnD6DrA73l887OkpY@e}ND;UlB~Wtlx(_*PM1xI;f72xPn>M zJj}-i;a*I}G=mVN`vmPH1xPpJ&VZ+DY)5WP@16L(mzGG+Hw91o-LPI^F-!kvi9`~l zq+>aM))AbrxBjG{wt8FAoklISyeDIFdo&ZXs$9vL{rar#v`e=I%W~0sT1q~2cjy&8 z{KFZ^tPj`|ckeLpi@Q?S1+yzkOA>TD8QTNODZO_OzKXg~-|A^Q6lPJA(bUCAh9K!{ zsdNT+$5pn^(4gVtiE1!EmDd-=P_s1XfGATD4?f+AD;E7s^Kn=C$k zo(~`%aZMOtqX>^JkLlj4$!bPRc9{6X+`Gf3Zam-;DI|;W->SVLfguh|e3)J_FaGxs z5ujKwR3xunkID<_wNm+LY#3B!V!zHOZQwy=oxxJXpf+I6lJj5<_%HDu9=Et<(T-_7 z@yvMyo>P+fj5#|y%L(Fd&9GV8SD^f%dOPrgkYNNSx8KXtCl4exj(_kVfW4A_rC^bJ zGIYrGt4ULFTX8K6SytZeuUlOTlxtw1i%mc|DW0Q{5UkQ1s1k+A0`3b)LJSOLf=ZP2E`0 zEP$GVtt)e}(}sH@bcOgt6FW)hsYU=_f{Ykt;vJ=M_l|TlZ+xyPh?)+o<8@gUIpV?SwOqW@HP~01Oe--7wpSss6QfAV46@tS0qWqM%!t?7h~nezBXz! z?~1RRe9C9+&Yy^p0pPdpYSu*jw_jxY{H)iRu-*uABN(<1%nVx_6bJgTsLbfd17xQ z3AzjNI5}J5e-W*FE=A^*<+awMJ@!dd(zF}gp4u@b(fs=&n2`Rl!KBPBaUrysIEFP{ z5idpW2){mVus6BOpSG&X4!%42=987@ly%(^4tF(ujc(<+ROa_$5spCf6??E$RrlFN zE)H?QRa}FMSg47;!%qdP1a={ghvP1g(>SE&W0V@k288`w0OTZL~tsW5h_d z0zs-jZdo|JpJeqR?ib5f7w2_GA&)^Rkc0y898`56jqcu=rFVJji29e1#Gy>?0(^O#&hT2_^twBLX{{DH}sjLzcVU?!&o+125k z`*#bl$^La#g1cr}FIS8K{MUs%ai5JbS00BLXj4()y|&y~X#1ZcD)2-j)lYp7XHRTt zqwFXd-{SmrQt1Q#yCArrx|i{7Z%KQmH3Lg|K7glDqfA{gx+4^;98cvfj%`e>qZk;f zp3+YA&{gcNU?!pr$M&tOuGemPVSl|8M?|lqx&K8%LeT2ymfc4tG7{#7zn4eMA4;gr0;;Lb7(a$Kb70(QO=kwPpYcc~(Erhwv&140 zm_Sg_Qg03RSoV`v@64(?=vd`!uXK*|Lgb`uv%K8u&KCKzkNpU+&J_&}YOrRo+D3lA zV5jngvjN$u5^auCh>x?-+C^)y^VfU0q$hi032cdsK@FPzXt)^6;si{M@ClO2w<0oylK%Yfw|JpJ3W~4HI-&dK=uQ}F zWJgEOLx2hIEH6!ZgAgR>F9l&%*FIMqkUQvA6gtNf8td-+0TA#T8u$4(<-iVGjtJ9@rX0+|^m!(2;cqdS9JwOz6T?ktl)aVlksJ5C~ALPI{=O$A}vc>wkN~xl$DGdxFC;}An^2b+Y zB_$P43{gKeHuN74vmF;NgPr$AR4puMq$IuWdtKL8Zv)@u1ds-LQdM+67lnp*G8YSn zLqBZiWdQH_?QXavC-?N({!?Z4-`%5)xuXBvag9)8Gu>LsFPM?(!tsH;R6Vp&fmLpB z5NFIl$j;o-askMbw|e}2+yo3wk@_9-u$VV6VuFW<2h48BPFZG+(y)3e%;OYq>+Uwd ztOFA4PYO`c!9e7EAz(?PYiRl)zOqvL3%`GrE$;c*Z(?#%J?}HVr-p6lG@(==GSX`Do#Em4 z_k0LHfb2lsI^$rBff01&H)OUmi)s9Mk>u22q5y5He5Y0ciB&^)@RePrP2GKEGU%+@ z@!)AL_^%qa?eze*r_1x4TqAF*59%p=?XIW;%^i%00<0aQ^@h@t{Zo0t4~XT;Z^yE< zDlQKw43+B@#Y6*B(J932xw5-vXs;G6ndm~n@+U1W?Iep})fvMP0)?GJMi+uTgj;zN zohTcS)vA;rT9-_tx{_r@l+N?@tPyONd|Zw+wk~>ESBG+W&g2kSS)~E%N*m4MhcI1; zaQYw5py4 zalzEP;h3?Fm|bGR#eYtLr$%o~HfweZsvNNhMU!Py$>gDY<2j5msb z)Gie2`s$39iMWV~{RW!)8WGy=Fg%u&wWf3j4o5rMuJ9v{>getEquzg{?CZ!}315<7 zu|SNp|FoGmX?Eg;gc_w0W^eMstMHJf_Q@D2xkjd9$H64nIfYO-``UI{jURh!SkXvb zOPMGbMkzK+ek!#qDO~HP|90n?cG6bf7?|V0lYPlnaAM!HCCQ>4i;6#2v@U zne~ye+a5i$bKLa~;aFwawH*-$d<*S@;`9N_#$6ap6YPWiQ4{Bab5O^XlTF>hdS_T@`%}N!K z-5W%nTjST5=m~)s>`GVWE)gikXGHm+Ik^@J@lIKp96!=<%|0wdg=sL@CO-%LCHv8a>Vy*hGL^ z7-Bbu{eGF{#5j-K=k|e})N*fj59}Jw6;9)wcvOWp5_N55ls^2BgE@V7ljtgDT!-Mc zWM4cDQYN>VQvB`E%_`ya9TFS$7;KsvVf;mZgoVs3ebNW??0K8BC?%}1T{E@iOfd2z zVSD|s-+4X0@>1kh$Y=DTpu^vIQIPDntCN15>0g!^0j-FoqQmFMGbKD>1rd+5-Vz@5 zmG_MNXQKk$gC=S7cw=hUNfKHGIcW|^1yE_hS5%zM&O0q39cth`=Ff$^dE5aKU>oXk15RNe+tX(Eo#Jmy!TXyL{F+`}A*h(82UD`y2>Twi)+DG2weo2c5>4 z!75gT^ug4BiDXes>hLLt`t#*_We2h{THEqQ^eGEwBs5E8P|U5&3=Wd!gjSyt)gEw2j>Q zUN91HkdO=dI(D9nxdiyKt0eGNdQXrx$Dq=o*<&v1UtU>BPAI;+(j2SnM144tb@O|u za;4>YMQ~Z~WL~(Zbs7^CQv3skX$X!Db2$d+e!uBfLw^HQWVd51Gf( zT-u&)8zJ{<>J+V8SnAfZQnSv! zs?w;{8e>B2;|D|I1u?7J7ovNepAUO^U!NL!c=zt3XuimBiehqJ7Dk-6CY3lex@T>k zCDw3-^UvnUz6iEjj0gYkW>2Re535V}C+8sk_nL$HXfXPp@{6+42E-skBB%H(EYUzU zyqYJ}^l0{Gc~(Wem?%G5Z&Z-dm1;PRK#$bMOIO6_c9N~dEu`$x?^xgx+0XRVVX;l#0tT=ofUdHj-6lw)W zqFMooHeA_UCt8Y&#X|%YP2*2xK#4U%Pn53@R z;S2QSUfH|Y$Gz#m{T!HgdHuKwqjwX27}WS?xNaXB%acJ=WomZ&Olgi$9^|3Ell5(3 zfo$0A^eJM<^nD+qpcKT2*&GD9?>{v(K~FxoEvm)0M?xtUd>OMpAa}7hP}C>U?W<7W zQ7&k!Q?Rj&|Djg13?*zt6RO9=y|}2V7)yfNB`{PtwuhGHWpz?j2+@o21w~GkL)GGr z#o?m7(0|G3*H_6q^us=v@kCB9GHIGEJ*FykJKDs=kDDoDS;IBob9@pjtJu?foOZ=a z{>DQzD2x0lo<|Wms~cSWl5d&suW1cWOHDmTUzQ`j5)XT`7kIO#-qz^c>ONsG_2eTQ zo{4j}he!7xqDO$*ihP4C_nBi+02DArp@g7(u)RZ&b|O$-fg*b!Cg$}ynwrOJi5x>d zP@W|*-lcos6zFzW%aODKJ0MK_(t{OBb3TlGNA8bk=myb!qRiHjd?TI^!|B zY#$lwg=$vfa1pr;-xrC|1s{gpahs&^2RGT^*A<>kZHqql>KDGt##VbU1({_2%kql@ ziw}yABd@aGU%Mkat$5er94d%ogg}}QF*wf3{jOOEb-{ClIeYzH2Df*o5M`WmM5622 z$n*u~f>U>#UaGE{VWsEKmrp{kN8hg{PPXe$0Xzws+Dwom6&3$g4S0yXEIk^(3f;)rd}NK-V`SB%ctyF z&gAbi5R!tSaXk|L`PJ31Rj71bMA z@nS7`VA|sI=j-(xpMu6J_Ut!hq`>plV5;X8FqkVvhOH}t;yBE8!!?~rM;!Sb)3@^S z(0*tcL1S{i<}xQ*c$G!C5N&UyMn9M+)Bo!RT#EN`_EvS-Dsd40JCXd5f+tFa;*$%j=23aXY;d=)jQjP84ju zDx{B+>KVDu!e6T@*_Fajw!sCbS_o%0huzvDf*b&af*BDrAFIC=UL zO|QEF#;@-8T{Oi{pnHSPFW)biFEtI&K7#X}5^k(fIhJtP2TxGXMz`AXxL^_+Tfsl~ zj_Zn(A9_|ho4SiCQ(Mxz6$L>bigW_r+5Td=yRx@swe$*B{tKSUPyXPGK=m_c6-Sz) z)G908gN3^Bf;CF7DVdNv+1+<}9z4&>kFjY8vS)m+9u&>JlL>LL>r3JLm(qzO1Uwv8 z%197h*XQNHnj!l`X-)GEF7-<7rLe=1p2tl{F-AD#hd8Z@#aUugZvcr#>!pcK^sRyt zSKl%VqQi;Mb$tzSrO!HDw_PB$th#~3e2)l)z;3KhjY^$&28G*F&^N%B9DS_ zAkV_ICiV#>H_0VZj2$Hv6+Io@m^DdLC=WLe zzB-V6VKchw^2#(IUX0k&M)b<&O`$>V zb)q!k^%QM}o_lv&yVph}LJn27aC(r=E6AHOY43cHT)(x~v~Zl-VY<~&JSlz20kSO>(f$>9!1E!6qzIFF&=2-_vNo2;j7Gy!bHue4cr#QFogUZHnXl#-*4b@`2pZ*>VR!qu%q-Qno_ z@}y8>17mD>Q6jWco9`{hG9S18vLCR1*`VBgMlp39z1wjk6354NUCmx5fyUha zs=ym*V_SOl4Q9!L&R0l6Pn#%Yp=AQbRvU>t{ z60gbepTuLm! z<}>b**UrhL1U?MgNzj!fQTcj$BpZT`P+x&PMynOs!Q7}Zs{hEy2!jich=^e`@&KN_ z4b)1H{cpq&yaV=t6e(RJ(gE@SRkR_Nkxe`vy00x_A<#tManD0V@7zOcO8iCXjPe&$ zVzxgF+#8~x!lKO|s!*|sif2&cd~)K0iTVna$=?O1xN#S;UX}Y}4WZh=3zn#QYW^d| znTf4qqfVKYto4fJN-i)-*tfi>G4^gbK*AvRosIc?OJk>mLHunu zYbuW&)lb58?1AYuVtea8-97(uSLk1QhK5Q~Qc@?Rap~X(;b31DL$*+it=_;7+&nxW zu8r-{!ap}(;Y;Yh=M8M8`7pweoKTG_-?IMFTCYc@O2Ed)po5hewTdW2vML#NPKQ*`dcykAO&Civ=kYTo^$sW;itkdTrJz9Ph*FV4M0*k1WPo-6D-TTMP2Q=Bf)+Av z>T4ug#&?`yNhSpCk9;_b4#huP)UsDS!_DUxi}(1aa{2Ec5%jd<@ioCsp*Dhd>JGQ* zDHDe6iD#)Bn+wBRovWWd^(Hb2#}b>d1WgcnCY6-OMz1Su(KM7fMIiCu;X`C1$ZW6~ zK0ACw$d+8dRxa#1!C){ah}y&@Q}y}nM=h9$A&{CS0IFyUKrWG*viQuWiaBOud~GR; zum}dL>F5xcpjR7f1E$BMFG0`P7megG@^W&E!uX6)XiH8XmLq)IYqdom_6?OyXu^FT zd`qi~u3c?@TAXouQ{cV|p0X<(?^@R?5| zKRXeA+U7M9WUR;=_(DI`)70?d*I)+_H`PQ&YuWGdCFseHH+~}Ih^|efu>o-LxZ*l} zuz)*eGrEQSR#X%{kM&Egn@6iUkuQXvPgX^-Gv~o_Rf&b6G#PHrM3W1iyrg|u7T6C7 zJ+8?3uN6(W7}jMWA=-q-Qj!eNa;~BmQcWD#Wva)ti!)aK9H44L9{DSqf{QJSqY$sb zg58(clUvpueF2EA3p9{jnFgRY2R(-%D=lS zW-_}p!x)CWJ+%%T)WX@vlRr*$Meykdc$(C%6g0H5QD{m*Cb|fk5mGU+*WIR??ld&6 zv*OY_z-rPO6S^p+F5$}$sDQ={NB2@o>wC9<`lAdnkl3}s516)ARLDxWe|!RFsy2rp z3cPw%e0^dXGr@lheUJP5MHTJ-OoF=!9o_7s%Fb5x0XVbr4C5$cJdc}+uv7^8Yu&j zF+nt*mP=RSX;hUSk z`id=o81(dQT^hR?C5=&;a$YE7RBNq14Tvzsqz!$pSJH`Wcl$mWw!=(5ix8N!= z6mk4+F%ind?u?_dJPDv}pp`WsH<;1Q`LbHs)CNZNPYHNqK+n$+HPQmiV|enN)A`TPYWH64ys#6-J1ibv#0~TpQG+WXSpR>?gv&CC;prwNX z834(1?+9J_1GZ$B8U<=*)pLwb2&P_W5l=C}GipD`f!wc1JgI`cN)0y2l%1R=Y)^-U zQq}v#1~C%Q34Jk9#nNtjXB zCsjEm${fVUU-gmot=0HO?QVkoRw9RmK(DRpveB-#CLK@ zJk>o`;d#t=&l_D{ix|G}k<#@}{Ebu;Q7WY&ofHVoL_1Dm6G8~88iqas&PMB-!zmj? z`mQ7n4vs}=iZR_6=P=obx04qpi>&vh5&kfx`TqA2jN zVX7G0%XiS1S15jp2kGjP4;UsVBG9ME_Nk{U}fd zj%u%sUdz?&iu_UNUN+kWdE}1+8^187*OCoCjO67=1Xoe;M`4`CK+#Z3<}pl<{|Uq0rkhW?wO5 zV}zflp+nxh!_MfJT+(YTw^RDnUo431N!POco( zb!IYh`)&cULb8by)~c@BWo_9Wqz|mA*Yd@ruC(rt&2`IHA~tkhNl^SwhB6Bd*|Iyj zA>UH+C`BTCzuO3#y zd5$M(42I=}LXY=^d$o?euCj-=BzhdkHN?urBx0h7aSqBDCKlcJ+(_GaKKkX)&+8S_Q`@+)1^*xf@PWY+d)l(8cZQ|Tr z)&JgaWvgRY7=*@U;gpGzR%pB$RwqlG;DEU>u$hymI36JRl^Hi>R_6exYU|YEl6!$j z)}<`eBPlAqn>@?};XB-C++t8Iz9Avd*JTZ%zz)}tDH;w*Qa;=kdS|osM4L_DY>Z1h zDHGRsJ72P*;XGl)YY9q>tCJ7X8?^(Q_ppp{XkDCW9#}N^VNfG zOv$TGoEA?FKEV(o5m6TA9J44|AHHrw0Ydf*>Uqeg@KOgiH!fhXWyL{;1J9m{LTjb0 z_IKFPm&}^7$$PEQ9)HitQ=gT4t+HNrU)vrUW-iDxyPT|j(LCPd!HcB644GQVu%oo* zBE`3FcV!5(!;VsIY!{80T(P7!BQO{)2Upk@K(xe3puh3JjH?+aKc_%Qdn0e!r5*7* ztfVtHL|6VN4kzb;1Aa~Btj&i{sOG#Kgh`a-VFd^1!n+d_a?~XCUcY`&9NW?n4u7E< zLhW;e8Iz5r5y$s}Ez2no3eUc=)9P+M^VRi6f-ehjq%%(C`5)**0mN5!5E5wr{=`>; z`3Cp%M#Js8OgeooNA&U}vfqoW8IubtoI=mF1C!G`rxZT@oV;#`z;zJTl zLl&WU#%RUlSJF}Db>7sV9sN<<7(_&;@`IkXRsGhM>gba)P5U2A)qD8woe$`Vb6h}* z9XbC;y}kqL8R4&GP5(*6^bq{iub})JMm;Ks8S+GKJhD<)<2~-{tL<($Gj1nM_I9 z7;|f&uH|qPS+7sXauOaKGkR*G-L237=iw*eIo|1eK%`~0bn^!Mxim&D6y17Wnz>>b z=`RZts7ictuiiOjdPJ$*|0;EsbMg%bLY4fh)06M1Szmi`C)rlA1+V9D+u$*NVZR;B zfDbN}=hvvDR(RxoFJo$i_h;Nk>@7u9X4u(;(t?75U!KXW)Mtl8=_`(oDBe81v>37e zIX%OnK9`k8`kxn~#RY}s6XJZ$^p~DO!4vjO2il1&)HB=}l_k?QVZqk3>)SR|lL+TlHuy7uPhrkZ}Zo?JqUbrJ}Cj1;{d8V_5!QLj)+ z95y|*zqVgNdghP}N@jMeZgOzs0-}R!0uE`{L!0d0@VB>alSffB>%QbNR0060@nk%H zEcqTqg--TGwr_ezjh~`Y+a++lQn$csTDjyy|QLduo!7Vc=Nsm@Az0`Mz;=67pR+;pVOYap|;icfnf?qV&? zuQ6Di$RPJ>UM2s&hqG8vn#K{IR?xHgmpu=n+s?fb`2w{wOFSPf5=wFyS&Y_z)JTy$ z_!SzCf)kH1864fjOx~{Ns?y}rZV-=2uC=97ZA#A*=j{W!)UjQ)xbg$PSy5)@FwnNA zfNmEajsBbhQpeYJ)jo^hB(MP1JCG@1-1E3j?<7)~^oU{iO>WGC$sS75fpTAW&eTMj z3nGXk{AsmlsL;Oj9d_dzhMl|3Y_HzKXWv1O%G2(m@sr|MS#b9ey+@o>5~!64NiQ!*mn9zjcd>lb!aDnhA_ve21eZM)fLup8ys)9mRlh5Js>2`DJQ>}0EIJ^3f>XssB zaK6A)xI{$szgIrl?(*yk3biz9&3qO*sWe^o%3*%MhxO5T6Ir+L*FVX&3>g3p7&K+O z`OC+jy@L2wSiRT#lA3lmN#%@zHhrHoe6Kt$3s-Yv6W26dfYNtkiduzwU@g!*RDg?4 zu>3Z8Sh#@aZL>4Ec1k3yQ1ScYk$g5Ug3<5^ZaSk)!i*MHVq)Ws<^ET>2ZC~9RyS^~ zNeYk{$*vGncbmcb0LYUe(&>(rJ=xSDrg85M=?j||z@vd;$V8winEFTy z*5%&)P91fAz(exw`*&I22_EL6^_Qwav1zk(1oc?mcW3>N0r@qDq0u+>rdP1eNahvV zU;;H|WsRa>U&yWuo#mIjOm_9TnhFp5Nt^x;&#qg|s4uM?^RGJpzE*%78A6jOiZknDfLOt+v_Jct@Bk21IaG+`f&!sD~+V++UB=GP}cf7y$>ZF{|r#{Na~Q&%w}mm1=ftCu3UqaKK`wtG z$JxcDX>NYSg`m!n@eP*|n%O>V>)5QPjoG-p*^qP8f3`h%WGRUp9?EFR%SLl&-K2r$ zuka1QHK0jw2-7uxtuQM9qO|5QWWyZ%zQLZ+XHZ$>nc6FVmrFtg+*0Wls^dr;wYXBa z!452(@bMReSIi-Ibf3BJTN>l6!lUjyCb&-ZxWSKRih6i2*_CcMEphC)f?g0q`htjt z?=1t^IL2VK676(sX-{7tT=HcxC8j4`8cu3BEt6mVoZRp*;G&0z zWsY9`mCjF|EimhOXs&cDL@#i8bMUuajp$mNK3K6_%@xPKOivlQVvi#QId8((}9CG_kM z-j)sn@t8uxV_t2#;$5)1BL|%Q`2GI3uLz+{467dTL~KPCPrUWhI&K2G!^ht|4Byf1 zh6>TFQMMO+mloT&_xb+*>*}`|1Dzm}8rNp}YTQ-mugU^SAaE%0Ujj!(kw>e@KqX54 z0efnN25Ic;%4JcLgr#*BxQ$drkP77OQ{;75LU(7I936LBi)?g>j^|_U2+z_-HF81+4UCERvYk>TeJe0>J?H z=tRshz5Zo)u23N4#%F^q0YW}P^=tnTlk3Hqyxp)|yCSIbkDMgjy$WGhkcE}t5Yc2+ zRNRt%qou9Q(MFzkUDP!zJX}mcH!ccJudJ;6wyrLjor8m1DncyBF4$LA^7iIKQJ6I4 zLcxI(7FE={tSrg+2J3=1Z=TjT&Zq%{o{W?S)E#Jw&^h6;-f+)-|ruI7M$>I^-pGLRvc{o z``E-6zwfd!MDHeHFJD~++hD$(Tt9}M(B=Kqs?W|C(j@WkrP8k7;up0xh0{pwwoQ?B zu#5TusxzRcRU_?2JGh`qaqW{nln}n|p7YbgqTAL>ZIw6+MJ0T&jUjOrAFg1NP8}2d z8oT>u829b>B;{!BlAkdLOS!g>ml~HYMVfu*kMaAs_f2xr_PK!%{`O?GGRcf^AYqxj zNh&BF82Rz>St#rvkyy9$S9v0+R>2t2x(^3wvlorCFDSL{@+?1-Yrsci@`AvSjB#zz z@X+;I-P7$yUHEZr@f&P*c#*aFeSc6-J}7gMPVIpbN{$GBl9dey?Tl87eG&WK{JR4t zgR}B%Eo9~i;(BV5%x#rqAv{lw(}EkZcA&$&?CeUk?f8^N%?}^e+`4uk*kVR0r73xl zV;8Wvx&GdTC;}Qqe1p4lI?0py&v0d?qqayyy%3j-ly5Vu2A!8fm6<5~z$w^!U^ zuEjv^oJ)exFQebd8hXUbM)1fp(Iy`t^(XOheXZXpZDz}~43fQ3mtF$6W#@i=Lrc9@9}p_2P%X-etv&Dew;quhI3ub>jl*t zs?!xm(T2^|(a&y8e%p2q*C=48&?TH6sPDiQrMciN+gAVV69Q);F|l;55(7nmWU9HI zmY!Ix9()F^iXxdM((7usi2HnW_H%HxZa#+kIhVxg24BO`k7u-#VRvz>(ssTJ14gM; z(7GmKx6Ol9usZ~BTfG3K8V-rlROWmSRAH1_kQmWj1t#@9KqAd%_|tUr8)kaCQycEu zNwvdw;ok1q&`4fnY|vAFC#kbmqbC;n zF5+*Uas}82hSH2wztf|!Eh1kJhhi#-?eMU%LMpA0W-lmtzcTw(WdyQzv==gy>>CYg zlg1pZ3eWgU4DBB5e;1-xp$~pLKcLf?9aU7n>bkLCE;2vn-41t*-j}>8V=K(ftsY+- z0Gdy%fCfI=9aaIGKW2z0$Mgoy6A$l-W8h`Djq65+P_au>hVU(6?TTH$-X?4Zn}rTT zqirD)XFoG9Rfxs*;xu-@==FRGE-Gn=L^aELy(*UG*&rm!#tf~yi#9`UcQk#J!*GwS zv(z~4;0}%f=Dn+7=b*Wogz$V6pMe&-q66o4#VmuR2|4WKGeQezB<(}S`VLV(;ak2M z5+t#r#e*XU8x!@ptphv`%9oSTO&M)8A!>i!q%21eSlgne@A+Lj!-R49eII7e=HJdd zGeedo8a$F?`86+gDFN|3G?xnhRq9O)_$?CnkTu+S+H;=p1ki{ng8ni2`i@PBbbfpH z%}5Hmpc{vv69Ow0egRe(oFs&^ahFKq49Q^qy{6}_M#1oeae#dSbeNTeCH&&3OQNtI zvoaK)f<5ADY7nOAF=&&Bts@hO4+_5(7y>C|p_Y;Bmc}7S(6BhCBo=P>5rQZW3=FUi z8`G$JpF1wR7_;U!TDLA9ne3nXAIu0136Bu7V&rdVsbm1uw{e-cDjUfZ{s>7xNB_PX z7`^~q0$wf8ybPbMdRNf5?o9{~X|sb=Fa>czy8E?s&mR#EiS&e_vv)+zItjc%pXAlG zqEan%>e9wtr7LY#w|yIK;YhWJLExAq;1Si*sH(0OdWXVOOn&e?jZ>Tw+|;~+h0ZMr z4-6KNFB2%EFCF01lV7A5eepAnSMhftE)22(gAtpGNG@pp(i0ajorKdH@A@v2WXgXo ziK*!BDANuzNr8W(ArS;ES06rl@`M-A0FeZdwoxCJ_Fq#V$~$h zT^r61rV@QI4+sefm88*+J-$5&H^WKWJ#4--1a0QszgoOVOnlvI@^%;W3Qz)F}xrU)D>T)l!PX_*q0>-QPm-N%nTpbH2J*mbg@$@jtz}In!*s zkG^X;R`^f}qfKuJ69*8IQZxZY(Q9_xbU2+0+68>6!fP5LmN<)(BzUN)$?EsOlh*jj zNAgG&={Zm>GrTrd^!D{@*)c4K%rg2Y1gDS<=mK zvVh_bZiX`VO}(JJHh^|zyM6nG?Vr~v{c4TK5s@nn*W{p&USoE5Zu_s6sg#flFAtvH>b>u1@6Pm^iH%XdEKn+Z{k zKp}Tk*7wOQkDq32xpQ3IB+rCB39Lu5HCN^Tfj75ELA*1)i8A`)|2mWOP~XH$b-jec zDwzoK%I)08icOFtnfx25Cm0h$2ZN9=^%MILP0Y`qKgn?k(p6y;DRm()xw|Q09Z)44 zY08rhxSiy9n(uB85qkcRI0k+8`IN)DAwOOYSqK-Dgi;|RCFL5bu8tfRuMNu??@jQn z6mx`Y2dXEL@Y8cJL{J=@?5{tpMKKrea;5EmrdmR3NdyO-g; z_BK)h#m{|RLdx4jk^gpd_6V-eMtPU!tOyOg`xn#R^{aPb1UHJAgG=9U2?ycuc5B8J zNPwX)uPywq+!Nt`sp6>g-vgB#_qq@?8fd1ed>6DPzC z=*#rbW)?-Z8#O-7<47E#&wWhgZ}e<)BRN@3CheUU#<_KEg2spXt!>V^;DWCmqe+Zz z76`DAP4-S_FEz4kCM+8KQmnkOz_v2G$P7e7ZOazCtUfs@c3E`e-d^5gfuXZK7nUrA`T?L}*^A{J z@L*c(lWpX_MYk|z6DH`Ju5Nu#5#}-oG=}nzQIfwRnE0qTCax5GIcVeX$C)cv@sAv3IG8NS*xpKwTquzGf%GS*ypj z3KHbq8e+T%6K5Xn5F|^cuXHkn(Y3kB*O-2f#?Xyw=056yirEJKTv`R$*@m$COTbW` zwXk4*g9C%sdkL_dYA<(6FuJMUj*$Wn9UMmaKKZ)kw=Fa|{hsyF=a*434Zjo}oaZGf zp~@L>a?1mRk3c?^)?$0){nr>H@h;W+6e9ZO-go?Yad4iiBz1tCc(qI7qTKEM!K7{C z7B_H*qu}-w+(sK{gfICRnXQ$j<;d* zBqSv4ihXR9KR07hc5Na=40PAQMUe--HTT@`97F4zQHOD>p&;pJrex5s@t+A&y&cxg0l``I0QHi(tphog%h*DzvzU^zVJyAbZ2 z4ML*^8sfy$zdP}p)`(7m2on#vMp#%7lmC9wML&8+31H`V73=x ztLjnr&0Ql5CK4jx4xinRIu5$af31Yg+aiqDFRvt`lefDZo<#zUl89_8_@1Nw?sWY6 z_|VK{@xH-ts4V3=bWfZu%8ktY;{@j;){zlXFIb>a@N5KaVL>9P2Z%{ zJyS)&AUVRYr|dUxKAaALj|37y7V$;h=;*7boqA3vyCEz8q+0iB9qU)SPTOJ&JtmOY{+L0eHu5(B$Xp{5IffJNQt`@ z)RHWRM%U);wJRC;G$zLV{ShupnhBGlWLwJ!#T9a-a}%-+-+mH7G4;B!QQrqsTA8A+ zd(LZObTkj6@~fuP6O34MO3__|_{r`G<#$gua9!k}!7ZUxVdLc3inoESDI3-1%)uBcke86jy4_=cbXquxuA&e zQDIn(De|?$>y7wj(B#qpUrYWNhjRo#p%JYN_wH3jXgRtQm{+r!G0{rwQv};;+Mwcx zT_i&t+}-^i(_ZrN*~q6Nt9c1_P+d>-HZ#g=Y0#q@n=$s3+mD~Si3&TV9FWym3JL{= z(>&FIi6bYei7?+{4oW89*Pv?6lElvRKUwWHqq^3j$BwZ}%;T{H3BvU*f)8c|`vR-r z`8cN>HNF24gS$iZYrI~1DU%SAp^Q>wmYYRf65?Z5SCQD>lN~=0EOTM!Ls!>nTxe6~ zSjh|~=zK=8Lek!@lkf)qjHH`zY2Z+u+;yJV$Or)ovx9v2`9;RP%V@Io%ZryYtx8a< zu$JfWpq=mGbixZPCMCU?)IQV^;4z|HFn~FQNjl--_+hi99fqj7_kC#I=_BLADoFnu zmKDVWxhvzz={I>x`?vN37#MFaK!s5BzCHh}v5MTa*Y&Vz(~i9lC1e6B3}GjlZYoG} z@apaFY~A-g(XW5`)kjbUJD1?4Pdm(4jaqcuG0MHaJ`fGkwxvnw zQ?;Q79T0TKyKoK;WkTEYll|V)G5@Hzhv>Ql+@|O0z?yT3>HyBB6Veorw8LZQBPr~KV zOHF;I%1nd(o|8*x`w@J+^Vt*zY^XX(cyd50sv|hibtNXU( zQyk~+LSIX3jQkod#yKbg&&z(|j6g`Fxh&A15%Q0CBUl z&_E>GeT+x*fNhXLLCRXpR80E3y%F@JQ5szlwZ>bwn1OcYt8%m8?AY9fm|~djf|n;g ztJ=2eQ>p-m;T#B@j9+^*kSq~r2I;`KPx$;`wIGw3ld5?>Mr4$3(>mvx+4NxRZWK1H z#^%0VDH9D1O>yM4<<~n>Vm_xJoFVJkjrdw?VXQTPh3+VYHp~|15GWuK#C?tMGG=G{ zn)%J3_SNl^#Ux`j%Sv6Zx$DU}pMgcl*5*NIFYVtt=7D|f2a2t@x^O4ukt)psy3vw# zL~5$z2VKK(SD3Q4su5@%(TjO)LeLDU`T{4rxRg{lj;^c)4}IY)PQQ%g5UOEdcF|k| zv#={Cuwhf7A$ghd*`!=XxS2oxIxj8MI80{0sIE+Iz~0!i zc;WP%pW*t7hjY!`D^D?Nvw!M6bd1ZAv>+l0nYXo1DbP zfWjZIDx7H+IL?K%aoX?~raDGM73)!a18~p#L5}avxuw&MUT;hIP5u)a!A@nr;;NQ|*nqR(5)ckqRLBZ= z_BAO9l*r!zRal-k?j?zXyhh79%-Z}4WlAeNH2Ni#2jpGa<(uD;j~09q$$Lzm{c|{V zO$LxK{0GaS6h`;ViJ@w=aKdRCbIn@_^I!wa%eK)rDY+aOEoYV z*3m2LsF2gF&>O;&9FG~l+BNxt`pT|d=XX|-yYN6MR?}chB)PVM6m-|pF)k7fffa3R zozlj0ka@9Jpu}+uBAf+xY{%amgP_%CcKRE{P;<^#U16`I`;xe|iP?up{K4@V#df?4 zNetFzpevqrI|p9CNcDtK(PX>?DmIlRHg33I8iA!y9N7J_S-0`q&@Oa{B5Yz@?C{$> zau;fJ~!m}a2pp8%6-C1C@YSH*+?RT)fTa$5&FsN?cy7gSE)aae-M~RCg zv)6+$NoMd?DTQFo8Ub}<(#F%Z1ag;>IvkF8KlN|=*H z<35?SUoUwa?ds9zQ%Iv=Ww5@###yWq?1HvM%f`mWZiHo&({#G^e(jJ#%hmKXg;rL} zD@hd?@+#!)%crI;eSmZPm9qoe9VpuIGu2;ih=_l(F8R31W^Ug6VB65hzW-jmijaN{$ZI@}!(c%%l?1xgqV^*U(_i=Tt^oru&fI z@UXB0ZG%Oh_B)^7A(R+uVb))Vs6B5wZ?&7pq|9s4I&zPu9gKYexfoYkQm zxGn@YF^=p*DZGvdgKf7=2)U2TY%s$p+)#~Fu%|RL_qUo3joPq9K#SNH?TDfo<+yc( zD{+uV+bnN!uty;T(40lVF@7+J2F)r@0QJYS3w3p#74*cON@!(^A4#-ATEm0o=`5%a z@L4ga$y+)FeXHr35;*L752l=je;p9Bptn7p3e zrt5vP@e)1<8y_M)AJQqzB0M>RXX#kI<_IV5mSTWb_ioS-)e~O)9sM`&_Uu#3?^w<# z##e5$xb**qTSh?}K^Sk1toc>hk2Z3uDCxMwmPpd#oIulu*D&lOaWSR?10jh&!_=iZ zQLZ#&7~1V%d|QF9d-U`fip!NMcb^LNG8>}I>Azwne|@wZl0f@Tgs_-09|^?Mp)d=a z`%@dki<+<%S+t_gvgdf%5zv*YL#v^t32ZSR4+4%FfK;|*WLZ)ZGovaV53CWZN(>(^ zOR)rWLgAytJRfaDXv^GKyEyFW0U^X(uueBBwid+;G5YF^x%kk}f^U5#P7~yJsM5^; z1B3_iMok+fc5fIo)0q}0L1fxeTCC9-&l@dPOOFKk`E^~Hgm@ zxnF|3z#AdcD4leEzY|xcCdj5m`J5D3;#{k)Hwq%oEdx?RwGY&eS@?575hp4&luh*? z=*M5{Lw)2DBan{;?gRe5)QEjo{=?^Os`bBqh=xc|7uQ8c|3Tim087e|cz?Frw+f>6 zd1f&}osaLw^>xt)h%)Hx#L$XBPd0_=?z*|%@dKTwt2}oep;}X>kIQr7u5jmWSR@(d zV@y7Na8>VcC>Dn#>K61xBgpd(kX#o(7WV-_rS9^(5pxMwkV6!8{i4)|G7Vls8&mzbH(7mVEM2A0 zO?&wS*`%xZ>;vOT|Ihg{A!CH`o@)^?*LGNjeSaw|i|0`-$Wm>G*VmIQZL-*#IF6R% z*tGw0m-55ju<}DEA=Nk{2rRAyE;mtPSX795ZfGaQ#@=SV9%55dQ`34*Fk$FH+jO#4 z#Fc7XX-R0RGW1CTBwR+r0<)3~oN-N~CNlyqy!=+ma2N>d@xI7X7@DVZ4Deo!$_5Z_ zEPcPVw*@aoF`-sb^bny95Npo?ec|QfJJ?%$GC|<}Q0WS0!%xtyo|o2ZA-2;pSq#5 zk|;xWED2r3&?N0Cb@OWg==2{SE5N0-)^G`pj@9JF_jfnyU7+~f%cWKV{~a9G>)o3hTqUCn0w0q-1NUML=*$$y4AEQ;nXX#5)sOyqN3^-igO}t&Ig$0cuJHq)&!TDQpIhy zqqM|WQOcb*JVVi~^eE4PEx(ypRFCOyWS2CLW9VDUfx^EyylUR%iodg9S z#$E&0sL^`}BXB$iu89S>QF*RQJD_7n;AnTS?PeQpY;VWPET$I`rJV`=c*>ZTcfkBk zuex_K{4=SJ_R&Iu`imOwf#~U0!EgcyftesEF&Qhblt4bm#w7~Q=iQ4>r&@{(bbiDb z9_R^;r8yX`1bat7im|?XDQoMT>AYh9al_}a_b~2XarOUbany3mh|wi3VUcHQ;yIO1 zV#xys@*B5BL$_I)1Dybg6rj}WhAk{E){mJuIhE-zhUMOS@ZkCIa;O|PAdBAhKlp03 zhqCQ}Q&fUc*&0k4!jnf5RY3{D3iJJgg9S4e2ep1gRJ)`@SHoOGH~_`FbD}IRE^cJ8 z-_K714`+qqKF7z$kME}3=oDmlXj&!aZpjPmKD$S~rb9(fMZ(=s z{WEZQ6=!^ZMqk4+rFZVcgVpAU}}N{y-1quX0P9$bUWEQ#%kNVib1Y9b8536wwTKDM@S36Rl19P3gQU>DNG| z75-u4P?BNf&w}FMIRKCG(66s1RNkxwiS|mG%sa)Q3nrS^(FVT~QuDv5my$Nc=%d@S zVor$QYFN203(egC`QqY-%&IBoVjuR$^=X3|b@{|RVKA5^>Um(`p~<`1+uQp>yZq%d z1(4h%RTFH@@2aa4jEsrVy6Y(?hs^*_Pu>O~m$L1Euv;jWtTGuBkBbYa#U=bvUwR9Zw(`ZHD2m{X zPUG=?)X{+<8VgHT7C%jhM9n_MO&a>FE8(U<&w*>2Q2M&1!>p^<)gt5H3V=?!MM5xC zMGBYB=F?%y4soBkI;e(3h=uxa5{~1#YWgXv3K2gzs?slh?99_ZA_$)G6;$#(WdZ!x zL$BI~yfwE*g3ktXFkNIqk7gJj3CHF=1)KO%h{{H#|hnOj>*3i(fkkF?d`kkjLUJ6=UTRQ@pZRR~SOSftcztIWa|3cAN z?NTGva9a0dKN6us41l{h2qKmWLus%KsjRpp$0k5)K@@$nS4WXWrD=e{ub3%0cPXlx zt;&R~T3^Vuc-*ft8Dc{u8yxanfGXdLYF33J6+~VWIm0Hf4z;C0p zZ=HSU+loa&nMygj)0@b#da_>1(k)^K?49blUJ=SypcAX+;&$_;{~gEvW8#1#Oa=u* z9UaQeFXh3goLmtZ*4j6Kcp&;I+mP%5pPl&p`7<^9kVq$E?yns-X@P3;FMj?>9UvFI znaJ;E#hj^PHFzGan)0V8=a>8P*E`Om1L3?w{YB#ouY!xyBpzPsUS%u4h~;`{NC<0v z5D0Ryv$0X#CIUU2W3OxCOrkZKODz#~jF%WHCmDIOk&}_tJ-i9305bef9tVY&^PPgg zNMeE0lPBeOL;%3hYhF4>idy08pdidZNhwn8fe&od*#DR*J9Bq87#(JhCnVHLBk9yG zEhF=`tSk=D+@!zQPz~}xx=k1sKy3`6H@xiZI00YaDk`uS~+PLxQt}zO(0#e`}3!Zup+Eh+t}}p{mh5&OjJ>k^7MXHyVe(< zWBu>KgG}R|44Lx`JG|fIXWB7Q$1Z9MJ~P{cil1&&p56%ljMe?E`D~GUZ$M;wD2j@& z_@qnjmgm0dnJ8!+p2 zJM&jhFLQ@arm0^K&{6QkYTfir>%U+SRzyWz|7i&RV%elY0GfpRBU|CLlZ3RFn0QnH zs9?vh!!~;p7?>fozE^G$fn_*RBT5|boBGigHhN|kx>sE<8s$R?+^H4WSbw|L{}?)0 z7S6f~!OPsm4_~{9HTb~xv&|n1r$ryd*rYFxOBD_#Qy1L=^Z>q%H)Hmi(vB(V2PM? z9t!{Mz=&L`nPLU`)zn>6L^*Ivx$GMMQ{?oo8)^WB6lBU|lX82kln5OTf|3TzKO!)> zfy#g1_?+|s6)v@hfI6roK-y)cA2I-y3lWbj{g|-tO+7mU2@&r-nm?sG`1-L9{K|Sryd3ux?e|63JdpZ+Q1)ILH{*}KOczWEJ@Gv@Edh! z?;aXPY&Pguv`-cNnds63bj&uFw>9Z~;D59%Zb=BgEEV;gefDswd4E{FPnNd!e@wNX z1?xI>fP7Hjw9eS;l@fJD_q~sQ{yw7qK}#UgfX#lH7fS^}#Zh+tMPQ-e^djHpRKVf1 z&Y6Kg*smQBr>0B+T^!ROpZ;%xUjDvX4*U z{}R-aG)e~973u9L8RWH;WYG5>diXckOiehP7}ooLD%Je&+r9Gmyb3+;{TFM^#%PKf{4xHqtD zYR}&urIr+}I5Y6&sm=NBdVciwF-4ud7CUwmNl*T=`Zcsg~_1A&==SLZg{7uu5!_FN8-Ou~dh1%(Dp_<138I=u;TJxfM9-MO2sQkNW z-%igAA5IxHt*O(!UvX(MHvX$9|v?ISH zE2-{UD;(?+>iY2@snoORTlr=#`TEB1)BW#r@JtFli3Q5)8|!IXt7)Jn?hjR)Q&;4r zDAIA{{&lVeXYeRt;xmzxJK(E{93DXwy{47ZEfKBq^O~5c#hXdr49f?M{<-|WE^b*e znUj~T(!DxMnU5BWk4|qiklqu_Ec|=iOw_oUHk0MQ$BklsKT1WTysvH7T)#QCd(d}T z@hH9N|GQ&g4!rEM2Ld*}AuNpzKX#oz8?KRsJ*<1w^!vB7zCgT3zHE=Xe%%N*o(wuo zE2Ps8osS3WDWypwrAN{qg@5nwKU55Aw;yS>&{}gy&COOs_Vven$a>XA&awGhWXp;` zy32|RB2nOcr5Pby9SqW2&&$ursY`9@47M6~@qajfEzQF-{VyWs^u`j+CdUi?j(8bq z`-w;bvj8df_g*Z^pyzTg%gBm^p$67M_%aj?%$AhXl25sh+CQY+I1Ksw?)*_Qzg5o5 zNQ{i`$tz?BW7-DJmMgdso5w#R-41>!C4|dEpcLxatxtcS+oL6X`uN=TDUKTZ)~xV> zS)Pi@;#dFDu)lV%>;*Y;-0lJE#%>6Qc{hP5=e*Bl!=EjfH6Y`G`*N*){Q!;+4=sjCS6ZUTJ)LG| zdPd}%jjF}cTfw0%(sSha=U9J_Tu6?rZB(+Uxi~Boc%D8J*YP>v@82dzU`)TVGoSjk z8$23VbYe$?yB%|v8r8>B+?U06Hhykn-ul1v+`sPu*xn`#M2-GkJYBNHQ|XU#J-No@ zNqsD+=NH`F0aXMzN`sqd&pfy9s7`SHDALzUvXu0E>^JRRy=;icAxk$CR##iNNTOqZ zh9;QGHQrSa<10%ij%+UE+J2Yxjd4oY;qO`gy_7%OS}K_}zyE;KlIg?cHaou>4T7qR zkhO7_7VOr?R~*Q01a~hZPP6Nh2v@n7QPZ5v?pkznbw`Q!j$sK$LXRM&?T%r(VLTfT z2i^;RbG(m@3;14aWG*_pM_u(VZU4Gu?jo{g zv#G9ywTIiq|4ZBhDhSPqS|)$wvxg$G{1-8hq~>ayrgzD&FtCHI@)Ih@Sd1>yevZ;n zyunz48D-I@xt+GL-{ZVWj?SV z0o?BAT`zc!YHoT^5g55<))>b9GDZG*IDs7$#NmHGGL$B=_k25&tAqFq(?PiDioro= z~rJv>SR@g}*Nk3TfP_5X3IvKeV9g-#fD4{8vn!d0ToF2L2ZLHUw0&`E1oXB1~+ z_bJXL45zXB%HFu*kTxSaeWpBGsF=>ETQC@qs8r&U<{VrPyk7+VA zf&s>B*E0vZbt<&wNDXhLY{Xj^sy|xwXKQfgL0{f;y+(3>PoZNb=(X;eOy6s^3cVB? zt%!-uiBJijj-!#jo zjnXttgl8Y~=$H%Js}))eUJXV`uf4!}f~OCU3_Yle91_di@H)UZlinP(f32--){cu&NHAijy7-0*V_;|a9gSqK_D@cD0`lVu-NwyXxs1Z{8A*<6=)t<8 zG4-wgUBOx!BakP=IW1mV4-9AC3hLS&7Q8-qKKQhGwE-NqN8Vs^p5FEw+n@e(+6Ia+ z6o|t%1=4Ag=8kv;gAdV=bkDOFOgrRJfiMlY0jb?{=+zo?<}ib~HywPKw_=#m>2cs{ z%A#VCX5=@nSWD+S5jyeWf4bSY@-$oj>sBO+OtoLGJN*s*a2~yB>XgeKuiv-{?L1 zoa}M=q-;7pRvfiKcdiKyd!41c1*9Il>X^^F330!F@b8eg4M{T7eHT}Yot0GmL&u!! zRRjhNPCQ+_hR2iZBG(~r_UDKmE0)S;*9-6X1pPP&8Bo_=USSQM4kFvrw3Y;1ayI9ok^()&g7vvobOHde}GqFr?f40I7F1Va67+D&l z7SHqQz2ra3UiGN)xSEGL4hvnD=^6M~$v9~^N1qlFoDPW?Z{_G4w$a6^OJ^`YCMb=w zaHCL`|1KN&o?*vJX10|p^y4c7ymA=Rib&)+6wsduhn+G-UU!AGU9l#c!N6#1B(lV= zj5G3@8a=V`I$g}X31q&UjWH`^sihWQEt5}SmR^H6H{DId1LV|Z zZ1@8pYNpB0Pv~8>vXKUQkKT)86(BR8)w=g}oM?#kd?e03`*fmVUoktKO5~Zs=jm<5 zV~Q*ixVZYQb&BHKvOTSM6Ih&jX}o>8Cb^{6-IBQb;f|twAwQmop|zzCQ|hu+xAH$G zDor9MOzK_BFCv4yHy}W;R%3jK*!P-4le+MA^@RZr3F8mF8G{;F(BQ+3jr4-1N{4@2Mg7`Xf?+FYtc;Y%>wq40)D)0jX=7SoH(o z;L%UHP73mPo``Lxhn9e6KJ)@t@6t*YJr75jZthH1FocQ(U6j+ImU?UY+FMLm32u$G z9mKE|idnLPGl3;)`WClp<<@gfSs5Px>)I_lF#7nepP@s;^>xC*r@*sncwaH+T>mqY zn{}_mv)!oQ7HishnW@4{f2wgS*KOcmipRvC6?yEVCak(FA6rbpcR$O0iiHf#Dl8-A zOJf{WYr}*5GL}Z~M}Ef%8(-AE=O;f?^!#+%L2Jhy;3FlTiQ25Ay@6S*V}yQzRWVo! zm9hmjmn_z9vP1iZF&9=(@M%LeoQAe7q{&~>IH_$`39j<)Y?=pepjv@*sCep$A^Obd zu1-BhOf6a<iy)5Hpqdq9rcsQrB_p3={gZ5k#&KZ0{TU0^B3|1awaSX(8Mn;$)t zUAwxNM4T%qbr6$KC6&>3jH>?&{(paVe%?n5@{UgTz=I;cuIkl2I)t}>qHnD~n zP4Dh7GO?!7%8VoDuS^{)0&De{*`#%ziPfOo-L#R1QuKWJb~-I!C>8L?*C9vCbbo;Ngf>+Ao+=TSi%zYXl8W|Nvr zi6*D*kKRsN!S_%E4-?0DQnP|vJ5yuJP2s$cN_xa<%yj%0n2Pv51=@Lv=$x21w;_A>CP~MMqx|0?S z7sT}ZMlS+C|9pPyg&=`v8Z488lfkiR^hiJC2|nocqndd+d2UrbGx38im7~+r{Pm-1wsnn< zS+KF}jE#z|O8tK`)D1VyDxB;Dz_`LA7I@LuE#~MMnNBuI5qR&C7pjEj4oZY~9;{1l zAR;qTymXbvgBE#!{g?0;?-?X)q5rIpCz(GS3HzZvQz@*U?eVum>ZxJ&>WQg#Rcf@p z)8?lbvH$B)MCW0-_w%9%(k$urxJ`ZTuTkve@a&5qF`9{&J*{i}RVm9~|4VgCIG`cH zga7o=HKcdwPMa(BCV?}Pn9vpA2pxb&V(|?!lmXm&bTjh9cu`9*XfcerLnO|@TT}&i zW#JW;{m#gJ(2N@;i@W31D7IWZ!T4vrGv)V^YLh6YxTud--%IxAjZ=nv$9A0s?mj+o zJ)iEmD4d3hfI7dv*rl-%$j|@J8sk5wVupeRMD(;yA93B{{oJ^bv3|8esPC3!t?uO| zxKY(O1w8$Gsw=|5Sw7?`xso*X1r^4(K*LN&?Jq%1X?W(IviE8G z@ttv+dAW*BzW6=nPc@e)QjlbxydX|z&KIdO3RhdY>|aW1&7RS%5_q;dIeMwRVq~RJ zFHD#=hi-a1BuYxiy9lp&G*^;kw#a&@zR_hk`>GtaZ1eeO2arPUUaM*-{J;J$X0S_1 zPgfJrW!}C%)Fb*=^_S}X^Z1G=vKMKuIc-Lt)hSN*d8h|8KGhs<^1ch2s02sR1h%TR z3KtsA@XjCez6@X1XjS}oQ~5f7*V>2b2}zbjZ@sv~+h=|sx4zr)JZ}~dLIG)yct!ZD zG!a~Eel0eHEJ`*yHKFjj@r$8?csAS)OIu>wDhv-nyi~G@CCuDUd~n;Uj6%KIxIVjC z(Q4#y;_#!;`DF&a1DpG?Aw9zWJ%D!fKlShb@tuT67pPb4oZp(;JXic|SacjFIQzZn z>bm;4`Cm1Q>zqIc8e91$?X>wXUE;exkh6a`3SqR7vgd(CBM`N~cX3JnJSBRLW4hTn zF2Ijtea03Q8tb6^43d-(2|eTzPN?q-L2iq z9;_3o%l>Ru3=dieh>3Zh}+;t(d?bAy+O1W+v`>H)2)OI~r&>h(7+^Sufw!3s-- zIUMVp6hvSQlHdG{Ym3$N3`9NuiZX4ub0JeIz9X zbmtty9IT?MZqvk;CdT^0z7g|^?d4AOChyzBe(SZPk`v6njj8X#u*6f52c@-N|9@V* zepY?|&6>)43fRqbn>M$N=p;0Q-HTX57RwYlmy(QKGkjhI8Q%m@mh z(h~7w*FhW;CRbUgM3gB}nPQjJy>||%ibYgW*cQxz&WXfb_CGI<`enpm4uVi#h*Z~Tu2VfGxhHa=9E(y-r}pN~JAEZFT`?Mlk1 zyl+%A;pO>wa*;Xd)qeSZDM0^SU1oYf=3*N!iOy5zB$*Q1q9kW$wrQ@v@==UMm5Vi)pjj@Npd~t>f}Ko^OQ|>d~C9G zJzozQXn~g{BmP~2CEGK2pf45GEN#hF)one8FW-jTU*nFxe9F_g)hxP3@C*=dQixi9 z_M$dgW#46R7p+ zVwqWfLu(SH^_EiNX@J87PSmkRVfzO*U;`yq@5l_Df@a@l5U7aeEs zA6Y@+CiwB1YOm}O#VU&U9)T^3Dc>$jpO)VLm!cML8Lr;J9YsUaQc&9{QL`246xtRz z;qOGr$Nsd(z$rArdu(gD^XpQ@@yqcM^ybUs&c5XcI%rO^W#yLT)m&?I^ECPM);AN) zKr+QRi#3ajT0A13y4qSGmf9C9e=UNuIpHz%fKVG%j{ya9zgQW6PH=a-%j`O&Y*QP} zx%J{l(X0rrd*(m@yU_BJ${!;>$pFtBdgmQ&M}{2zQeu;HJ&N=V&r@%mF8G-&c6GO* z=GTI2-;m;V1kjJr1y!+rHbN|Zy1(&ayfVyT?z@UKWbtnJy{p0v4Gl6#B!9V?{@m7q zWwv#nHpfJNf3BX3*u=Sn1T1!`UdWGNZjy^2zYg~sHuR&XDv4^nzVKs1T*Ef9Wr_dO z-zWdOwU-U+U(2qbI#_?Q=ovlF7{}dsS&NtA&GX8HEBj~5`7+aTL3Y!JLfE< z$uAlYaX5E09(B_Xml8tg+_{8HDrW&PHM?ETd0{C{N5W{FtjB?sJGCq*7xT(iweiPhtN}`JDe=TWGSP)c~n|$7;X&m2KV-?1dNu&3r z$>5>Ou#2=J`D}E5P=4@yhSWqs$)D{rYoMXF-R{OGbs z@=eq8cV}#JA`izF0|sFLn)8vr`s2qg>}KYfL~6q;M(`kYNC`Z(*Q!WSnxS`fP$gTB zwrzhytS0r)8FS!sd;A!XvM`k}luf$1&yO)c!A~Qn(F<1}cZt?>oMF86QyMWDs>A*w zT2FyJ*u4IG!E&qY??Yt|;sn}77{ZZeJT)w5Wpeg2JA?c_&|N8c>9GEw49R6%Xu_X} zGt-2%^B|u#)A?WZeynl@_IBXQryfX>P;PL_X|F&&`zFiC?7yt5UIclbYIg;;A48Au z@}`%*WIz9FS;|WRLM27_S}8>iVj#>XOrRt(%A`xR#>f(*mm+&QRb__zdk3rs!-C^g z?IWMG=eUH#;B@5(+!Pz6ReY^b5uXaPR$?F*s35^oVo4n{a8>b^&il`cbpM4iU_BW< z86mCO0hX%OxK+^G6&Xjy+J&P`Mf^^~z^=X%hTObo;qN<~#ln}Au-cF}DhiNAOERUX z;GiIf>Bi7MEbFVQuh7U3FZ?R;7|9f7^2a*b{b|I`=ZlMrG)xxKO9ZKH=4?8sU+zZ6 zhBD_6k{i=XCS{nXL{3Trxlqwx=gX>z;SpXN0E*>ao$!sxIuq}~E9YmP?_AdWiDMt0 z9H#<@K=yR?GjIsY)GXP@l@1-&>06L~VM|#CETCPSJhhwo`mg*niO))&)Yo@S4)Osg zk7-W!)dw+TYTno!a882cj8M2UeRMw*x#GAR7{O%(-yT`i0?4vEHa_S z{O*Y8Ii@kjTk|ojGldlx@C~V)EBqQ8Uh^O!dUD)My86dEmaWt8SItCl@RSlX!AeEy zNb4Z;N5Quyp=YF@vXxZQjK`PK|J;ZEC2OWUV@>Ih5FJ64$dwK-)8#$tWf~d;PqUaIUT{^5mBt3N0S{3-x+%dsj)3Q;YF?0c@OpNauT3af{Z*~^PBdGw z_cI3EZ$b2yLlTmwPO()1E@={JT#cpbNLnU9V|~b)Yd{LfGdMw}HgBp*f3o(+to%cf zl6#j*7|SY+>oK&Sxy|r$TRUB{gu&t&#b*(K{x)e}_ul)T_RVJj&=UcyQ=TFpFLk7M zn-(=RHszMZ-^(n86j6Kfogw~IUs1eS_#0dKtWOPwuUU!tTOS_&r|(yN2L}Jm)Gm`P z%mTX=99-u*S$m(IEnsl;t!VT53L07Jq{L4QJ*@lI)MRskjq<%bD7?l}S$F5*Xkedp z_#hemQYSq)o7O&SUus_Hc>Z5JJZ4>`vI{i2Vt*JEc@1}U&l!l?MR&1HVrozOB8z@I z%&DB=W@%&U)*Bz$WdXN?et6p#Y|~26Ea4_p=axfkj^YBOD5y~i$lm?B-PQ~)3>r-^ zOZ9iCr`l=p*#ooPYZgi_y0deaKsml%O zRzi$_wwCe1M1t^1=BerPxLny(?Fl=~di&itubcZI=>F}ikr5u8sHh#z=qB(;_&u$N zXAdRjhT&JWQQ!t&>_;D1@GrU_qcLCvK0)p#$PVWOdMg_VB4Qx+Dgtg$Q04F-InpAM z+|&0c-54&P5Z8Yb%J$ra;F-pV5LqExHldH$NvKqxUUqO)2B(nCQzlV3(qMhy>#H-* z^k*#Ta~N`!OT$mUXiL||Y~F_SF`|-HK)pZ!!(;7;6V04$RWfIoR?8A_Nl37=e1Qk_(t$W~7UrPj15t7Z|1jH*MbgIOKJ zHUSW*VbARHJns)DsB%r|LMcz@YV+($N%Cq<$h=9FK4@W^H0u2yuLS~<`Z4TsqPy)G z&NqxNtE;i{3n1N!TS7t-AJC+T>zjyTX93fTi8Zag5W!_SY2Prtv~Z%|^um=;f3NvN zKjg^FjF{m5yWFAlU+xUMJq;A7`B1Q|f;*l1oWfm@W=KT4^3t~ z#TqQPcMi(%2#_t|rC_t@PDI-5ICH3Zsu8z=|1G(}FKFSE8#v?)eeIY2wnQRY2U< zP|u;^+xwBvpQ|KCKN(uG#STJB&`Ob<3>tLRA7sn0tNsu}hyBW1Rv)bPOWV%C-vzWZ z56myoCv*Ai$>f`5;n$kqKTU-#`ztYAxrW^VS~ZH-PU;B<_uTc5Z;yD`2gpyfh;sh$ zr$8{=2VM zaWaG+*@LiQ3yjwiU-=yh(6DF0F5QCX$A_}-aew^Iit`q$O z+-pekmL!o#Tn6|ZCiquQ7QcHBIg*MZ@HDJoqXK2y1d5oV9z@}4QuGq-gnmKp7g<2- zJZsF%WUt?DKdo3;3`-S!f^Ow_Ly;u@ie%(k0*|*rxrt`r?`#!bcpSpDzhFyey=Gr> zkAK9?5@+mLM=Z1^9(_7i6OPkdxaj7fXD=)!I5pW?mzwuOoD)J6<&kFeA57$dcFTKE z|9k|C1v~l@O%JQ^m$Aw**N1j(0=_j&5tJc}zOdkxxJQL$);(*X*E`IpYF$>AA0<#s zeveA*I)T39)xm`EX$sy(n_6nCo9f&~Qj(wF;b-S$e>c`dq*~N;+HroT$KLMH?@4r< zbUz5fyNATNZBhipHY8%p#R{nPRP4C_+~D~GbBvMl&5hr`mv~yZD6Y;Pou4Mn2p*## z#e(8GSc?ZAA!1rmXVUr8N*pwgu}y5qYHcn!vxbu&xG(&rVQ{a>u2adB3#|0zzU2-B zEc&o!fD^H$&^NU5SGlm+TvR>Rq7nW}sq8H>RZouV4KT^jMI%+8YlB!x(Y<0PTrH7* z`JsiyLbq7yy%Fi8ju5n}(ho%{JmxTHDKt3q9Hvhd$<7STJ`qKg`PaHAU}VjugWZ7O zha*4s+gCK&^=wP5TZp;-q(I&)ez-l(cF*RZa_PO}Q2zF|g7y9KFaJ5Ef-l&rGp3Ji zFXJ1E+m?P(Fc?5JQ_9F@sTJ3riI(L84h7{{K6Zh86LB2qwfc9O!|&pmI?kLLdu}|i zlp%ra=2d#Kdoe!8KD^rFGj@puAcI#FOnvke&b;#j|4wtO*v6@56SAzzG^%@4;dAuu z3|-EDQVD3szcHFF!gDUe&{z=Sn|3pr;OcI)BxNfuBs02UR?84Qbq5J6aa?5Zp52w5 z%4=spayyGis4kKz%vPJ_8ld_=;LzU3<4X(i<_6lRr)cn4x|7kko0=9PO_5N=Vh=~1BAA?wQ-d@ z0xh)?yIw=xRrZK_ohFV2%Z!-3h3cY%a-Wj5w+CCVNIGowJ~F*x^mZ5SCiE@QnJ<%9 z<=I2;MSNcWCGyN?n&KrDJNkW8(;=(gZJIhWz-=S~d`psQ+J^p;4)Ph--g3PTtB$N| z_tVd9LpD1F|6nsWtztE=B!2y3@Jk#rQDECgj0sF4 z4#OpJ=r?KjQ_$FrXDvW83l>NnnGxZ_nqE0vO!do+QeUFRtHw<#U--p{9-J6$Qo#r& z>p+r;3tirZS*QKbqY^8PrO9e0-g2~6s?A*y8IB0LmmsOBnqA1K$Lvrvd&f}@lCOvp zqIp(Y^J^y40QEq3^6gz8h@kSaWAFT{D_BhAw;>SJJ|nJ4LtyW*q2k7r-eckD{y3zuiM) z4m(Jwn4UajQPk7fb*eA$do_1}R1gXI#8Q3mQy%I3Yk4%?!|R?^Y@8hVTxa(rt9tbl zz@!I!NO}*+JbvhW9NX4qdKh@+-KBN*+9*1x)JhYR-xke1z!BtqfFF?7OL0X+51J__ zmQ068LqwhtoJDZM-3eA$-F6`)J$^|-G0OZ>qV{ODl)&Z5wm|h>PZn!#Lxc&C@d_v< z(InHQy9LV_?SxqK9#67QN}UcwqV|#e@!Yt{%RDH_(mHTC%txV0XuQgF5--e=G4KJ( zfG*LB?=;{w?!V1XoS2S*Pc|9A99>?}SYCyX6sF;AL8EX)t$Dx+WT)0r6&oSWM8D zVFAsOPp(Z12Tr&wVM_mM|KPf4U6o<}6M}MB@j91N`d5ylK6B2GuvsZ4-az9^O+yBU zEx#I)XykM`B4u=V!CQY-W}PV^0~MgmgohjkJX9R(y8rOfiEq)@B=bJg&oxJKj)ATQ z?-@+CU-4hcdzohJE6>;@^<}mKv6^tB4X9rbk=!fc+0EQO zBF#G2B$-Iay{us$T3vD%V+Xf*B>`yG7zZn(K^}_tNVh}YiYd_+WUKDZmT3Wvp3htb z9ux2_tAmJfP=e((rN|oi(Y8vvI+AS;AVe^;#pqpP`Z@M2jRr&Q&8O;RafOTUERkQ? zF(K-ETOLR@l1HQ#RkBPhjTo_6XIfab1~pMp*m>H zoo;1^+HgHayet(SLEP>(*K$L&N$b3t225@tC9_tCp13`=S*d*V_syFXn%b{@5I*CHT6^07Mwpg0^$A&&u_j_`EIy zPIg69{pKyG@^j7Zh7UOd+~X3y545%SY}5qWaXFL=*88OG!s}Ik%={6aenxIjb+S*5 zcmK}bJ^z_#Bjer9$R0M9X;>u8o3!fflt3tlbO>4=WYRc^aPns#h&dVkcYk6rjeO{Q zuJ0?oWaawn#xH&3`iuWE0wPQqGMl3c#d~#}Hh=wq+*zTDqZ7LQjwSDoIEyf3demRQwG2%d(^7B&EUxUEL)! z-QnVFyF0$Qy0Swx6{r-PiR{!0lsc)Z>J$suqlA&8Elccx*t=*!Pz7stM zhi%ba6))YFGU*&Q{dy}8!-f0HSY1rjZoEc+NF>T%o_ANz)C&H4IeOAZ*#Za-;bSJv z&oClS=f{9}@36l<3CMc6hBguCDJ(#>>B_qi`|SqKBv!#Cq*s?<*RG=sOuGaPE>66b z_DJmkrhOj)5cS%pHSryaO^vqa8C9^?H;nvb5arVYzw$oH6Jnq3&Fz&4Z)9e}Te5+m zuTF;hVonvA>?ICTd8zyyV)LS~7OnG4&q3St$piY^YXLn`Tz%)Q!hPos8?7$M?G;M} zMG5of-HFW#T!~L6$#J((XEO&;_Kd)fMYIsq-C6x_a^P>EPBg zH>&gdPe=qB#`%B{Q*$Ej{0*`RF1b^5HHV#yKW$`zO8n96H?LfaDS4$wD`&exBf|^< z&#C+Ff)CKNH)MG1MGBQ%oYKoJ!q(DM{iS0Fksrt1S)wCra5JVP9ykv^jQPyaee>u^ zr8Jp`C=DmQElW{8WkJGp)cXo)6&&}7I=gSKS!5&ymK3CiBX#L!Ey|0eJ<(f&XOA6wsmw?Ws(6tm)TGPzXsioUg66nzi?+j0!!5U5BE zPwggLdb#$xaVzwv;)${&qu;Gc4yhknw8by6b%xIsk~(E=RH&njMotUT`+Bz^lg&|0 zQO(!=*I!3cF+9<*sEu4X5}v06lfxoD3U%;`2aFF63IX$e^*9^TFAs$Q0`%J+$vXX6 z46s;YJy>b-Z|Hv=7jRigUR+A-SFbODHhqK`Qp%}?4`@EnfNrNep&k525b0-Aj|#u{ z%S+@igwyv$@4H-tW$~8WPOJ&2nb)^$_?<`=pJEH~_x%)!zSwZ$p_?R+>s3n#8d5S8 z;YM6Gumj!x&|?<;(&FH~UYfQz$MlQ-o0RSfxJG*zin7$zzH?1jm-p7t6Iw|8k}-vj zV%Z1Y*{a$iBanCPhmTo~G!*Ym`tC+XD(2{DX(hF5{nI9UF{dlPKN@c@7dw)No(_6! zZzzjK?#GjH4a!Sr|4s3cjv@0-`~|kyJmYHh{`UA1WuF4;va09Z;EO^|lm(+M;gN23 z;^ykA?@_$`ZZ>!!af=?@l;HQ_qlR@5su+|qKEO!K5F=PcSU2)LoAC#`HwTc~NCPY5 zzoMqI%yh3HJsKq&{@tC#l?D#B?+i}1DBCIpWuBGsMd6j%^F=s>@EK}TWzs?K>PwaB z`|!IyAkQ?IU5ZVHN)9&!#|<~lqoSH)jffRujePce|3Ch9ae9cEYkJ$O;QZkCF0Qc& zj<{PgYT}3bWcKFm`I91_GfI9vy>ZN#<5z#E`_t=hDy^#bkH?UJ!Ru8LUFMVt8D5Tp zrzcBl52}L?Hf@JqkSKa(=GciLz0%ZpWCsnx9uqP!JuNjk8CTqL5KYS9x$f6_c6z)3 zKo5qtx_*CmVHNbYwEZ8?Wc3ReO75@bo^4lV1{tkS15+jwjQRFYVf|DtgFP|0BTmxv z6DfrN&%$Jc2$asqr>|+xekF@8mjn&O^ux@@0z@f8d0@9!S{JM*ikRv)ot;K9BiyMS zl6T()IrC*!fon6v+B2t^Sa=6jsnq>@ta`;~-fBupc@tS+6#UGNJ6tP^Jh%Mt4LvBT z+63dVQOK3@>$GqLC4Y%=tUFZFYVn|=jCbcvnw9>Nq--2@{K|95@Lf+%a_~CuVe`*BSOpz$CAb#c7*!n7TY#vzOE-nu(uto*Rk$BjLGN6^oSF-ixS+QRHa-WH z-@fT1oO<3${QUj<)A8`|upyMlEy)`zfc~}ezgYmMD6jcA$zHk)lh?Qn@7W;o$C5te zjB?i%0(!})Q#cjys;;AD8h=*x+C;J%mBz`tfuEt!i zsC9bpWXaBd`>4eX=h{q$+y93F8 ze=?5?i#E>{t{ywIiTUff9!q#F54^em()IVSphv^bU6umZ(jbrW(wP9P^!+5Yv+9t; z9-FrSQrom$8}}QOaF|Ar&06~J-T&@X5^Vr`Z{La&i4snKyFcE}6`SsZ4+3FiYu?vBq*iYU zBlMlm%ADS8}gbT;MmWE1TQ={H1Q&PnXFMEZ(Kqj??C6MV{ zk+GLqTi42I?=O0F{(3NcFAeJXa2DG`~4C`>w!Q$C@ow5 zSnHT2tp7?My**h}N;7m?;%BjIuw2M?RlNye>&R zUy@TX^>743$&|M*_nWaKMQW0-|0Ji8!!w@&b-~*R@{=r$xE`}uU}QTGfS0kXZOcEs z)@>#2GYC{6e~zOX4a*glI5I1`LZSB8m6W0s?8Gm1wD0r8?3`@ACQknb%P1?FC5iIfL$ralF@zM0+uA8-9PPCm`f5rsa}}7h8=musAKZDva1Ez1`dfyd%Ha zRR$(GGW=~3S*_UMlPk>>GG~jFgN}uWV?rivPYznMN}!o}~Lizid@} zBD-Ujj@RQ4^!3=4?4o*4Zr)gADkdGB1{w<=p2#`9%wNcy)<+$gwPUpIa#;TNaN_EW z`al7BG&QCX=-UM%VEb9|fMCg%&gv_54{#%_eOaCs=UOc>2tn zC)n)U7LS+i)6t#XulTh7(0nLB(UIU5o|m_vWkoNQz(Mg-{^+Ih`02TiBVt2z`%thV zswI`NN@G}I`aS919$GXKZLM~Gs)m`tP5|)1iqK;}9<}C9tTkexn8z~I~Py8e2NRbJu;UUp21LJ;)kptqE)INmm5Lo1v8McQXBBBJR zGBaQReUQ8n)h1J&v&!ngoQ@NIu@W!pCS9K+acP$$!R|w>Ns4(xNHK$y$>#NJuDknR zt))s7siD^02*)jvT8y`ny7$UV4-E&ct{mHycM)(DoqAS8a9Z^2lM^>3E8L5!?|xae zU#U|`kDDK%u2O<3sPiKG#UtY4j+(t71mJzb6uC#;i853Sp-&*rEXiO=39BrsOip}= zp9zpzPfLy3Xd|VVBGtk#@({KZa(Z!L{RIPN*?*bsqA#Pq8eXE%5Xx z@Jv}s18MX1IjV1VZRy`2?@}yz6v47EQfwzbWoe4W8lYiUc!QP&5|d}DV@M#K2jz}r z9W>x(1T&_%St{53-Z}vIGA`5S(2teQYOk6K`rP|uE_D$AZuW;BMDMNIoCom)qinQt zwxymoJO*C8|Ga;hx{$(r>!mFgzYniq@VV^_hjwPUKk7L1`j)Qzxm!m=kGgM0C!cse z3y%3k*WJRBn%Bh^UA|k?^;*tn3lL?{?|hxRv_9l#liM4 z&nH|wbx}n^&m?~=@aXK$$ra{!g0ghgz}6GAg`ZgAyEEi1UA8LWB6*_hBF{Ye@!~80Zg>CnSxdg znTPM_Fh=6%!blt+RwMB8LQ))g_(eQ=B9`FK_*6F6hsJX7i_)E<$aHCz6H7##RTzD+ zJ4{>$g1~fFQz<#35?`8#P+O02h~kR@uFu|z0h`C`%ur}BwIen9P<1DcQ_VPgvO_e* zl4GSgqgJSs#?jY9m1$<%j7Z_&$sS>^6w5Q(vmZbr?qJfd9Qy-%M^*{k%Ikel>Z$7T z9Ncm0@s`SkcHSXD1crtzvXncaJBWzZmx$ex)ybbqP*oU<`KpQuBAV5`!RK??)0^6 z3|PVb1VL!JRp2Ssv4ODn$0=Wl)y;m1KhXFRZ`|DUCDZc(;IZkB=_$zH6O@+{FE(=X z3+j>j@)TI}(G9GSxc}JI^|gP)s?~OkLcK-@?0j45To72QfW@hq>8}HHnHLZ0=UBb| zQ|Cw7CwavpJH!(3b7s-UTgmoNa=Uh|3;E~|$JuJrnTy1=4ylX>C6=N@5x*zQ#NPYA zjsN%HQ1}x_ue*;)D8fSa2IMw8T)mpNWQgt$2fbDr=i1PU{c@vUd#!-P{v$Bu}&RX(tq<6I+o1fUXTLb^KQwuH~7_xwk&=h`3V$Mc|!0 z{ynmS8Y^}>EK2fJ_^eed!1?fG|4_r_=T~g6Oay{Hz&uzfQ;vGW*y0;%8ED!D1T0rw z-rOnE5D1WDV3$F3!k&_uQ@_$vM@Tj>W7GI8Y1S5wewAC=O^Gz7EcsNEXA(gDHG-LS zT|cpWob5#M!t8{44n1EAxsKUZojfJ%%){n~5w`4TB)8ST!lyTaf*5h^Vfd)P3g@^` z{zpk)-z!^z<;yvStSyEoYZLDCX@Zi1Gam=>#}&0-P}ZVFsYlhbI)Dr*4uL?8$Cj&S zf8Cm*E&qAVO8%q)A0ZL--E#T$?4D)cB8!($bAjxr=b{&PZRnw_E~e*kn$KE))(y_D=C=1)Z$jLm`tt$f3Q;UKX=IFbu8U z(%O_mmUuOFZMS5fP(PBR9(-rUv$(NBXhyeLs7 zs{;L#?LLx++D=q)a7WS|AI@^G)KfN$eE0L(9tQ)`kjSOb=yZ+{kx;?dy#p(PNwPXMrbFYIcsuDp3 zNXz5XXO-<*7@~UBCcr!r2bK;E9EjjJGo@0pq~a8_XDUyArqkQu0uLbaP5-98Iwh5r zAtqaVz&in%Z?w=Sk622NXQBGaG77o+T-2eBf$erC39)6Et$=eI89J zBV2r$5PfXmesVXaoCT1k*a&u0-#o8g&Qh}sc-78*O>=Skv?s?%wXh+(2{sY1qjdOv zy2iS&d&wurK!k*~Ey>s@vDHLZra6GGZ*lktKDVu9cmnPmweQumlsewdS;JKyG?E>9LG!L(yGXh?dqs-% z&dov4gjx$N!5S4;^+e-Qy68*A){l@s3-9*E;mO0sj#H_qzb}Ic(ro~&HL9lZgpe{o zUmEzsMegb!0WB>!%V0w=ur%rML?neP4@lV@S+W!mDVto4p=}{lxw)kZ#O)GA7NrlP z4*=l2WNFaQXCDMWo=jSH2^0MhNBuuTQO%YY9)yzD**?6B)XYjUMpe@Fnt4xd1Ss+^ zm4!tz+|Ij`OrczBVy^5i62{TP2+0>_w<3cKN5Hs3IS_@&QEq)uExC@@Lk`&>?v~I0 z4q&J@pjbBgLuJzkDMZX zuXw>@D~ng)X$iBWDveI-Bfb;@lqp$~AY}&wDTs1F2>;_oojc!rAm`KY_WDOUZg5`j z92Uyu3EfhLkxu(yvID&t?h&U8(d!e72o)g;?*ks}l2uarM9;0{OC2@V?a_OXsh--r z|5HzDh+9%AdPUWbmpYm{0r=ZNT_3vac6si(|Hx!j=Src|b$1ojpZDviX~v@|TNgm7 zr)jtHPTnd}hMDB1M``RTO%d~q1&aYTYa%b4&e@8m=q8sRYdsm+~ zUh2@PatQ$&fDzMw6wYN8InN~K%*yM()ITn@3~Xz|99w@nY*1F{D9Ikrd7URTiJ_6o z8J$q^iFSP~F;t08q~}sbWney;a_zLwABfDo1ne6xCOIA^n#H-nUhH>-6`mVbY7KY& ziE%-;yBCvpoACHcmy$w%_4mIx*^E_teX)#nXP;*yAeQj3ghcSfOPvs;L}*pH@AOE$ z4b{Qc2+-dg>9+tYJy|-*j0VoLCwI<9$^IJ-2i?GX?K~jBl#)i1Bs%Ba*Q=SmV0OW zlF7|ZsLZo`j(|Z3Ff=g(Cq6h{IA`$UTeim zf(a8b{U}`Z^8i0H-%{;arNy2hq2tuU4p;Erq)X=1+6CsC}|e zry@c{$=8P3dvP!+;(YtMy6@c^L)0~0H>Opn18VTk=~TxOyobm(Kdq3hqAG>^zgTUV zLEwhIrPz#pU!}LIL)2q@>3o#79AlH`Lao|sL*<9VmE-bkZX%vHkLH5k=P%lsn-)Pdo zuXeVTv`{yFT2j9t;W5AkRaS`q4e;fxN`NhFzAP5uHQvS(d-DJFqNaWX=ANBC{StRw zs&?i<>a$B_g=Y;}OB~A)!E_d(r zj7DkIzGg;{|0<>cIW|3fVL=r27jO`geW#hB6qq~^ILGicmUEgPR1pZeo1HV#zB<4A z5AdJzOUxoIQll3+w8=#DtG=|$=T^7hWHN6b)8Ntf?w{g#2w`t!`)8YP8g;p|;8i3# zrNqjS(%VLVKEeyzOTJ7q)AQJ0s{#?1;-!%%|IbmJQbRH+8PA`=nnt~vmP0&|_kj+u z{rCxjPMX#+E=^svC7)N%-^=KD^ymp8gm8Sn&I?9kVrbfmZ*R(9 z_l8~5A1A;%#VbSQ3RX?23P+nQ3`YBYLqUtr8v!8x{nfrl5GSMl=uUG6qNvLFUS(B- zLDuuA`v9YLYmC=XV@;P?o3|}ik^a$%CRB%#s$Yp+-TMagJOA?!dIRi}S8k9xeq}aG zuM)?l9=LL;TOvB9-4leebcDb=&lG7&>)*6z-3TZ(-JrXytA!euA^&o{_J-g7863CN z@Z^Qr$N|h(fRB4jQa3gSzt)I#giDoPKa5WKdVV+5VOHTxTxKu@{Z@N&nZr>4rE{yS zd5wGeK*#s57s5+2kwk|xPc@S6 z#QpfcIc>xX%Z}{*Cu~(>;kKv%vMbbL&oQkPZ(=f=KO)M=Srnp7ogK)LS93)LX)8`S zkO!CIo^&Gg*$P<47v;v^mWtGMB2-=g*)Mf zQ?v7oR^L#FiO=#Pd!|$)ZlND;*siFb;;429HZ>?YIG{?yE0Exa=-H-)04p8h=1Ln5-Gp$d{+9Xo6HS(bYmZ5sAm75 zL`Y*~(Z9aqhdO=7X`MQ*foADIqn(WHx#=CbG%0{3_iZv=6y9CSy4osCeO?}0!;Num zYS(!0EzUuu0G+5eSH7Z!b7F#ZSs`6zi%>-h42t?YtNkCKN|EEM*8ms;TZn2{!^aul zZAw8ZG_mg)a(syn)XS&t`GU;XGpXe}f(*T606liTNCgT_u?ofuGjE#|mcfgfMN%HA z9LHtA%Gl8CE{A`P@y)3~-?vq9e5YLbq1_+Z-@n@mu@o^*f4EeKc8y|cU)kh5nf@B+ zk*6`U?ehK@N7LkL6_@yxZuBY7=gc|sgX70C)w8saGy1s=zA(o7Q`O;0YaRCv6jBdoHRQ9h{FdS#=_{mE!xjNLs$Vc9!h}3QozwUy-+)!2;R}juCGC z<$N90zha*{{=}8GrM%WPJmZ-WSHsCGhAncuNL-}JjyHB*kofk_!g}qDfoIjL_=ENf zqXd`7!{{$`=^RSBHZN}6Fj_8FWxHuT8ITB|lZgexi_`a3vV3Ki^*6 z7KtsrzM{RR-nG>pF7ef!`Udq2PCkul#q>3u*@TxF!?EGwRJHQaY|6DWVFrQ`#z;OonZU0w#PiF1D>`D!4dN5Yoq{Ztscd2SA8!Q{u$q8Jq=AEgeG_-BE z(WjL3F+;|LL*`P$+lf|F=#Muz=o9biYk@X|0BE3>jx5qNJXHAgP=qITDzejPUE{p{ z&T`~Bx}=D?O1MHph?|2QzYM$DsWy>vn%9%kyDC@_L!HK1u92AD;u390tt68wMO^XQ zVwQ9uY&u*n+*Gkrr}Xrz2S*x5a>ceAu&c%Usqpo6tu1vW=4h}jEuvK@0@U6*I%va| zxH^_(3OjJ%+lI?s$G=^GMm_2~`H9Mn0}T;p@IIKMw-$yY}Zaoc1!q?1r}b zwQm#e%bzgI9(+@i%PdRvd0hNsb&exvpSxh|WG+kl#sih`O@L7=4lEln0cHMhTmY6h z4lRq4+vz9}bx6epz;1*ayqW__Y-vJRM+b2Hq|B!X^;Dai16yi`*-x$=*>}kOR}x_M z8<^X&=Fd?RaO!*@25Ei}F~x{->Fc5Uwzb)KJVB{LW~w`{x~G3)DSW!y)tL6gEv=GFG+1m9JP$ z7e6J<7DeG{uw6pv2o`xlQUDw?vw&8 zqCC%?T1_D=|FI4!S;3E+`;m<`tdXe-Q?{B*O+6=^rkj( z-%dz5ZffNWj@`c3di>WzdF=-b&A&zm9w>6+_}--Oeae>bTa_v%m`x`hu}+%dR#B5G z37fK|HiRj*c}`vQU6Jmai)CD_rbAZ30N!n6OR?i$Y)npbs=PqY481i)uGRH{{k#6Z z%CP&fWp1>x2R}K^WwPIi(&keM4#-vfZA}fDBWLS+)!3-27>3xVJdjD-LIQ@Zhwt|; zWO^wrsME?~R-&niNj@p%!f0cQv?65QZ}yhiE~u-su$A{tR2(k9uo?bGyC)W2AkgIoY`z66lv#5a{Ch?TMIO6G@38JC%JC>{ z!?_F?w>1Wh(NflmS5i_*;^l0w z3JJE8IhZHbkO9WTJm0wIqvRVU8O1 zDmsjpq}n^EC@y5b&;_^Hb$SYv0rRC@IS725C;B5_xz{EAlPxBJf?J0-HlxU$uQj5Y zDbby2o;ZHHwWMk_yZx1F>2_()%@>X-)%L=~?^Q-K$$000%a!?_3XT3l602f;`Bc70 z==qb1zYih}=WAnEWWdIlaJ9|aZnTk3lXv#{qwa|u=R6-0hr@KLis%Lb0qZic&B z_lUzl8?Xo zGt8)m&{Mj7z3mx3UupMmzqr0WI2L)Ai|2O|L>|6*k-|$aemn5Ns!fSjtGNmn#{PF1csf-PgvbT2ko20!E`Ueq;S94N* zTmF2Izx-uS?^eg*FgsEkq+&D5|Tby-zriS-R$*^Bn)LpCQRMc*fwC#5U< zkC!T<@ggB*#IA};&ONd4)5U95l*Ua<@aLn%5P;DNejGUXG;s4{f#nO~?rzs;an7!y zE-zeg#TEVU5^$wOXI7kJLrS3wv52fjLkuM^mjp~gm6AkI;{HKoC`;Z9tz4`2 zeIIVMW76nNaQl*mT(rUW$Bko-UbXSs;dhP795*%>9nt?sk_jS?cDF@2@-4I6gd zVz^;AW!!`tT5v-T4ZvZN8o?mL9WI1{9PRHPWOF)$2F_5fO{P6CM`O^7ztBB&;a`2K zu=r>gTH9DZj?CLcBc_iY92#1oxS#bTP}5zgs^i~~1lrkc^j@Ah)3NO&zWs+2epG=9 zSR$jj9O9pym_M5EYCfyM>Z>^@HK?szbPg+Ft z5pyK3Q7FR>M3rl!N<{=vZg>$v=KdOOPnYEf0qC+f&FiufcX9Zcs|eftJHK1an`+Gq)hP2fcQMa5F_*g;)z~-po4XRvjcwQep1dQyi)g=z zxbxz8x{k8=C8c6R&39lfto zmd+!DGqYmv-hm9MQj89pFg2Tp&n}zZg@w)MR(A_}vxXt68}m(%c+XCTu;}%M)J=p5 zQ@(U0w}^6|wYw04WTxOTMgT0Zcm zJys&DnXz;OhLYQ*IrLAHzxt8wE72U&&L z8YGhgxafGXe%8*|1*T=I(e0r}izWZzu+>+T^ED8nCrLJbzwnx}9wlgpoJ!8g!S~k= zY9cP(WFU$VX%|<#M$RXq;PUiuyXUCD$TUDs7bz486Zm_FTW$orDab?VUj{|yKH280 zyV4?xlZ+DuFVI>rWA9|eQ@{E;5ImY5BEaW|d`!}F9B92b$ARMj;k||h`39O9F|LM9 zRx0!%V}&`4j}%6cQi5HK@aXCSk4%&c5C}9S3RYLoM!G=@vYRZ4M_vLsKF|3gEP_&2 zoXVc<=9W7pN*Q+*%2o&DN3Jl#Oiuzux~z{GvVm1eTa5c~QD|E< zPi;udH~pJu6Pmk}U7jOpWnH0oi1|8V<;`>}ZF3rLL=FB)pTp7gE*wGQgCIS zqP=%g%`b%O?bkT`uigDw{O{}ihq}_M#IELz{|G)HZdcZ0pIs3qg7B9QPg4?)s<+F# z;ssi7-ekCxE4y-mUU#mq-M6k03d8n)nLX{7KMa=l2jPhqau@`9^1t)4>++p%*MFSR zpIIwLzHGm;ZfAa25!Wa0i{fKIG^^N~t5L!`Gh{lXK~4`>Mq}<<^ZnIUphDwEEXZt^ zag5QhFk>1*niy4DK3nN@s}a55$(HTou+F_8a-o_L(6>muw0fCH>FtwoA>rHeinIN8 zF-zg*+wRtFnOYQy_Syp`Rmy>&*t#kb$_VPA0*={mscp-P>eae+*S9MpVquM8W=)K` zl#(AXCk5I_jOUdXrmmr*lg5-%&LiZd0~0YueFMud8w;ih`rVn_?YCR>Om4ib&5ZyKDZVqy}#V{NfbNIokVSBfXUv#--@Nw;~Y`82(7=qY zkLxjt0~@p$Z=G_AM~&@pzQ>SW3nSPCH&_9i$z~u9IuZ14!9${U^2C94T}$&n~A_v#QWM?m|PTcGqAD!wle7Tu@fDAKM+lgnHvpo_4tMyMMBbR|f3o=8%H zL$4A_+{V&)(e9ED_*5Mf!c=f0SZIQdM3qPKmZAVb0!GUEgY~ulybid$F;8}5Pf;Hh zaua4!J4bkVo;J##{*-SCAhyNK#zBJ#nHU7(#YcfseaWf18 z8`Ty*hE)%33PqjV8q?gveEwSNPZcdaJQ8A#*eGsNZt76qHiw@i&W-LszP!^3r0?Yu zG)O-dsl-C=?MzV_E|yDYWG03x733EiCV>XEI!e*#-#;r1ifa#ffj z2@jT`kgs_+^2Vb5^OZf;CS!VU8(|ya@3Dg3Q-&y%NM>U`P~SLZ7x$98YuAjHz~=m( zTa-+07r!cGi%gNg8!c5xl#SxMy-F^quS}@HL-YBE^P#2?|CKYBs?%e8Wc6k|JZ1Z&4y3pp97zMIL`tNp0< zuqQqcvVB5nY5f5#`BF--*L3YSscPs-Y1Oa4?{y-u%64A%7KNq3RB49~uU0?WWnO}uYH?lJ8zyX2vRTQV4(3#~BnmHLOkF#TAE4N@qU|I_Va zswS8A+%PwG#yMhcDB4u~Z25sceUk>Xt0gCPl$le1S5Ntww=Mul*DRU)PatiIv?#h3#*h8NpPN7i zMyheM2us0`TxfAn!>k=()yx&*eX<8^F*DA(6-OHp1xFljfLukIzF{=`Li@ckVDI~i z{9TX<_n2dg(~+~8^UO9H5@pl?1UdeZo+n0DhA*PK*-%#wtrfj`5>AZ955M%GQuXP~ z?~(5a-Fx<4=3R;00E zIudZA2zs<%lV+!HE+Ks3ft#5%O8Jg-6~9XVzhOx`qWM)?QJf{f+l=^``!yF!cFm=KTtrbRHj<_gXxSCw6(7@lb9!?(*RJQR zk~$;YPTre;lX2XmSQ8#1Rc>1Nse??fwQ}WT5o{1>5pNn0v0W&N(*-F%r12n4Bl$G+ zTLrRU7I}HMy*_U2u>Jek+3$0*blLcKN_5_-Q%jskZ1pzRRGmr3(ovZ6XiufXHR8}- zy=}RW@;3Nr5UEXSd~0X&AYQPmk=&@!)T^g5KYV)nt%=ptcS>n*lmCsh!=W<1j!Vu7 zPbA?m?&|WLFv<#Y$0thGkW=mvTw_bH7IoZ55B95esfYt1lwFI9?6-m3jv18&m$2Z0 zh|*K@aEa&0?;D+2EM=RByhafT^J>o(KMBmwn>r7_)AWPSk7*6L`lJrA0&3s<-*iqL zF+y~*ggMrEx-W%83koPhm660pNBZ(d;I@QlnBJ{TjFFE4#c=RE zV#}|6(S@(f8>AHQ231Zc<*l3civx6ZjNpBOeB8*W6S&xLZy|<^*o7=Gc)}gLoIBOX zJRe9JrSxZ#c3x?>ueNZ-;)g8Npv$Z3eqCmoF7N((Qv^S6)I7uePM};4|KQYC49=I! z04qrEau)VCb^!B!NX<@B!h+Hkd%kqkAhDu2$O|%#uo%so&u;}+-kc9rqEUN5s9KOW z*ihZ~#&dW<9fvGbVaR(46m)`1iaCS@^&jj;xd+W11tGAkxzAB^BG1s1&tk-+B(r}! zZuZ8jt*U4*S{oj5BQKK&o^nEg{xnJEBJFfRQL@vTOH{t{GCZNvkP1YJx-fg{1tYq{ ze?DRF&SIKTe2d`KUk_3+bd!H#^R!+%;L1FlxueDjGaCY{5-438JO?>BoXCgoqE=8i z`BIdnD2H2}Z{m_@3qwMx|#7-|`Zj9G_gJ%%_^ihJE`7rwOC)$6z|kcVm= zdz83(7Tb8TH@9q(hpN@H6>E3IH!-=Ypt+fhGKQs7L($g7wjQ6`B9O$Pc(ihXjOdw? z)XVorVyE8l)-^=ycI$s5ej&d`Y3?KN+smbNmvRvgjQ-!!@CBnoPyXC1a|qE>La^kR z6-n}KYSYz{e^qv4*Ztj_heBpm<3+-fX$Oj96BgKapwze*S2W=m^CfaL*$E+pfw`oF z#j~JK_z+rNKFyX$qyycTj~37V;MOgW8gZ0XAU+ zE|fT#bMl3!ku2BfyM0`S(sj)iXmKPV1(zD9H1H%XF`CDKdo$#fD3cqfTOd;uC9-p8 zAJPemVBIi*@B$Wue4K(sP+MV}e*c@OP?*tYtd9k@>^1^u?5Z~VY)L(uL58BG2Gqtz z=GZYqnkll8lXwD#P+_&444^F|7_30=8W6h?Kpcgh`Ft0h0 zj_IP((prQGoB^K6oHuq(yFq~V45-3SzJ*ka&2y*BJog8dqEYzePXbODx>3k5G6vjq zsm>O+_qA6iviiAk!)I+)-%OHEScK45z#~AQ{ID-#8a01msFtAa!ig}lD@jg=fivkO zydfk3j5OUay2gex?03UFJq^`HH%l1_x)B5lA5caXP>3MkI#3e_1aG;)c=`3^os z9pJJ_<~S~#@6qj&T|W67+S3Y^$5tS1cQvH%MFrR`t=}7=w6jpH+o7~HOZu4>sijdl zGfe$G6CuW~G)TMD{nB7L8X62-CX=vw`@6~vQ^$6L!xD7-MjpM5(8m|nAqRKGrFx^g z)x8D1%~_FT(eWKnX0U<;)6zNf#|oCmfaJC#7?>)-gb=nlP#vSq_W32O{CLRz(Svl# z$X{a4SSKgOAmD#Xdk6%?Zwqg?hkOF8{xSj>a)0Tmi!!ZyMY|Ms@LiNH6vTpN2Ysi2 zKA-1f>DQVRdJFInWy|)EvQbM;!(kaCja^J}R#u+4U<(sw4tZMy*&T6msZT?-M@BtR z-^E0+Q4;lOz{-Sqduu2s{rkHVjq{J$qOcBA6da?M z$2A5Xhe(A>NFBtBAEwY>X&8)X0Prmk^3dK3$-Zluy`589)ODZ-C>J+bS*`=r0h;Rr zWimw!cNrDc*3Kw@QCD9|hJOI$Q&CY_nldYA2%8q;0(1~a&htFn0RH(R%#hTZHuV4p zI3-u4s4n6VS#s8UFn@ZtVXN%yA?g8BtM@SD_$BtTB=>2 zpIPVoET-pTL`;1meiibqS!Zed9-Lk3dO{^Hb;x}NY1F&eG{=i^4tn!*lNIPhY687k|*ERrVI*Z zeIH)KS%8eXSmgp#e`{1LFJM{B>7Acl5otG$RvgLW$WMyv5wBsSGQVc6oemGRC74=- zZKq%$%Kuhb6iTP+ac$*+t0i$kz)#&&%#b*IfEG;E(%?1jN(JOS<>zH-f;$6b_2Soy zFY`Nil;%lmQ+IV@7?}u+P`h}j-~U*OU@lMp2ux$MXC;iKM2Sz33WBInP82KE z01Gh>fNo`(4%1sT;?`te%-SeFabLZ!3(wH8-qoz*=bE%qKCXb&;=KVKY8NCA#SL!YMt3C zv|~Os+bxnNl*$|Fvur%&q$xN^gQMA3U>-*vk1vv9(5b!60k^mams$ertOT=hp9n;Ij z%@_&0`zKj*^8yru&GS@V^JVw;dtH&swdZ^PcDZVj_4)Q-tumj$<6SKqZ*OmQ4QkGU zrz=EfM2{F$BiF+*RuXtd2>r=u9}wA@W5B;{lqyv7_h#>JY_`81tb}~?RwQXKzCl7# z96BT4*6!W9j!MPLWYu?mO-Q54I>WdmgB|W%S-)lM-h}yrYc^%ok}>q>&gko!OVfG^ z&0N=YU)d{12`8R!+ndo86-xx(+HEFnrU1ji4*XGkZjc$*R>;#tb(!sYHfbLkHWzgo zHCLciSwA+)nPXo921B6BD5XmR)n-|pMKd}>EhtW#D$-mIcgBce4lHk{L_kJ|f`$Bl zt3(M48R8M;Qw|_aHF^zC93cv#Wl(_ss?&_#8}R)0v1I@x3s6ob;ajYFlSlgPy!#um zs=y?E!bN}yZa=hG4tvTBIkDoaqt7LTt7AFVVJ~5nQ(-p)QvDTKCu5i ztOiL@Ym1#!0Pa%HNX3Izy^?+5Z^&PdN68O;qd))_9Oa@Nzspd=#{+0-bTX2T_45&- z->}Eca@vA5II4{)LoIc(u^X_88}yB(B0MchR+O6Z=DS{4M6ypnP-#)IsZ*Ab`1;bU@df%Y!L_7a z{zmqhm@r$PcW-xbnWcqE81Fdr7(P|_p`0w0r~>X|Sq*8sZc)O!Gu_jNX^r>(Qh zjG0Hkqs<*y&-hT*wmnsm}Gfu>?d8{K(7esBeyqKl*|+JuL^SG)b{kXlf5AmILEkfSB1J zmT?dT4OHYD3QBv&1Y^uUj2XJ?CPjaXMl`uJMt{EtFatU(@X?uNPwHQ0FBQsgnffY) zna1S+D3QY@yKu7k)d>ip6(GE7tD5TM8aST!)>Yej2u)#`l{v&+~Lh0f}7*4&>AIHmCo>3)mszVMFEDEZ-|l z_jNdM>i+A-0imKHd|BeYTEdy;Zaj4PpSo5s!AQ4BH+)h_s)J9R1@z2ApZJy4^c#laEm{HeBX*DTHD+ zLz`B(w+Ta=ZAq8h<`%Db_2-SXeS`wZ%+0qH2pguh=)O7q`pNPTHs5t^MXXmUU!1Xb z8+Qy^dJ)!JVH-c$+#k}#CfKIN!%?Ea}W z!Vi>j`ZiBzJWFEWVtcAq%D8MihPi*#tT0BOj`Lq{(9i~^-4i9>Unh(07@jlJ7ewpZ z5nu9iQAEpiuI}5JNh>B46Bw2tp*sJ&D71kahv%2WIm%h=Pk~mP$2r}~RTr`JwQ_ft zcW(EvvMvCxtb9Y#-ntV}5$d7n8@qJihtcAL)eiDj*K>c0ltz#OjPftE>&UvDKYt|35DP>@QIHx>Vi; zVQOykeY6mE;N*1aT|pbb>`^kN-uMK@iC$Myv}M|?^Hm)jU)K8*&}S#XrA zBD8)ruDXoXPti7$CmfT~bkPXwMpRtL%oHP$JJy1jR)-I(Y6txU<)vD-Kvdc-A+|Eo zl;;l+;e`^OT>qObBG{EeaZUc1it}B6Tt@^!7&0K zSkiF9tyl6-VqC->QI>;pBck#8wjJfd7l`BkIOo68iV%V(7lpQ+a3aRpnk8<6Br4h` zrtAEUul>dl@!nm5oj~k+s$lGNiL_4Ypqr1xo%$N%_)_-!5% zLMuwoP>NCEo~tnFv4NB_8#Wdw)m`EN*|5#gNR zNxgX+g5&YFul=sL-D(mMQ?TN`R(mi(_(62z=>+e2-1jUYtL8xDToVU__U5*n%B9aN zz(9ie`S#Wvx9P*xX4^M0WcZO;7yztT5#facnVN6x zs-`EZ6Ug~?Qee1w@0=PKBQTg|#2FxNG6WZy#Pd`4^DPf(bsA6_wS8AS{_ETK?On5l z!G_g1-h7I5_Vqca&Mo|AvE-`w<+25Yv*Y(`qJCH<9md=X37hX7;rrtCwZ9qcv@b)c zzIWZ+phAlA(6K8LHKj>>)Sj2{>;5$Z#Va5Qp>qjJp}c=>)KEPH5X`w|#;S*epAe~t z)UgszXfYX`#(=Ro=VFe+LEezcFs}NH9)>!V46)Xg?29A@6cg{GNCp~S55^I#DRaLh zf+zv!`Y(Vd??In?UiG9R?=g*c-{b+*E6L6B>A6UgxuDKV&~&XzbOLf3o29BiFU$Mc z#^FP@V!UyYf&wfq%wmmafHp%P&(xnOnYnRC?JF>aSUG`ayb(TxE`;L{%j++37cf~; z2YB4H$ucY-LWf<5_)qzHG|{2178&cy1f-QCa3j>8VEUI;UAIi7^ai_;vN8vP2+ z>za!8mxr1MAY{OGC>v;r0=aQwqa4%=M-jhSKhu28<=TE#TfPuMxAo#iduLu{KSKZE z2LquFZk;{C4hSVdp|ooaq=MCc-`sBPpU@H(V0iev3XB{m!jL&iVm2Y{S6~9H^r%Qm z1}y-&GOQ^$d-&~^E#@%(h$2$!&wugYl?4@rr$De`er7%+YA`HWWhKBk))c}3wBlrZ z*&J%=WcAyt>+a5f>vENvLCk$9q_-QZZ-Tl|!z0>yo0gs7pmPtQh zhK1LTk9S8#njS`kL_$m4PPP9vjvK|6BITLI>0ON4_tu3oJUjKjEIO*&;@bNmjAR6O z(3+dLpJxT9J#p!Rv*%*xzK9M4%V71j|3&0Q*asN4v}cFc`NqU6etMbSV6}7U0NBAd z_ok5WgI7|2Bd4>^~&ot-csneha5DCSWq&5m&S6&1*BZGaP6l zR}kM}82I9{!Dd|7Y4=mn=En@&Kp4=C63c|I^W+i^oUZ|rj>Pk6 zv>^AvE7=mo9dq(t89d>>Za{@{(7LVlYdWW6s09!qHMJqfCCEn>v38)@GSqRf8NeMV zbuE30u0!B9*BPU0Oxd_ov!dg_7~sc=b4{)&F-l==?+h@By&^D!_zQ z|7Tx!gfxVr0Vxb4Oi~Yqut6(f8`g)8B1FYx^8i>47jU$bh;e)xK{n?fC?V_NPe<2H zRemF>tDH7pG^RfFSPo=qy`0s?&6tdP!M?lwJAZ4NYyKL}ZnOGXhI5xA3GhV@u_?5p zfssgoo)rykZL2^Y?vSP?VWiMkB#4b@RRiF(xH=3^{=@0+^Wc||^1ea^0ev-pj{D+@ zOAGVi{mJROuW*DIyKf3=ZN+DMnyqg$Ef#7hbB=17uoN<=pfR4a93W|}T3C6K%4Hoa zb7{>j7*Q8x*$)B5;f5IpNUWxO92dRKOB*~EFaSdvc*_jDd^srwvfpJNL3>8_l8%dr z?aq?1oKFtSkr5)=zP|#gp*Y*uq}fA*?E{vr29f^jcZ9na%dwV>7@0^K69u)tS}1~s zPyDBl*icDSTxRY=6TZ-~o>hm^)A~rq<&QOUe+%#S*oqtb??M}Be>TScEAj2#Vzr!} zk-4vD@Vg&NDjj8_@vjDg(N%H(6;99-oI5I5vN*@E4t#XpA}u1ShK!J8_k}@n{qyf5 zOluY)^Gur#e0kBn}6pFw|Dio5lz#UL{q;X?5qS%QN=IkX_=4BC^&~!T9A<; ztP^RnrZKPBjfcQtQGONNaVX-O*av9O?8Qi376m*Sfh=ydz0eC(H%4iQ10!p|qs>4j zj2Lh-k1ykNed0JERrBizcBnz8ZR2ps)Xb)CwV)f2Zj0Bck-H&V0msw;^onqb*j;V~ ze1`dj;!-(TM1bN!)Bp>Pd`V2&Na~k}cU6}_IC-BSo(Ni7aio!nYk-LqXSxp^RZp{s z@uW3=OJk7(f+TpZvmoj4;$Ec^!o$#-h1`U~58}{Zs;PYyNS1Rk1&HDUe)a3u>*}fP zix-2a|26msuo5|rA)6A)HAfeC8n_PCnZn?3&z*AqK+|A>-gD5byC9Pn%;Q3p z<&t4P?CWtG6Khp4$rG`cn?8yB;1YEnK4%6WIyg{Ta)r$7}!`i~p0hQ3NS+GRmEylEDXS-jQpsI^7x@ zAWsaMiXfR8?EWw|KzDxrY^T@S$WPleGMgrb%E0!A_ZL-df(pmamBy#tT1Us{qHT-I z&UrqLwbeuZT7Q3B`BCLkO1xJZ>rQO=fQ_NbS{X3egyzUsxpM6;w_2 zz5784Q>OPR1QkNHA4!TOIk5=3y1YAW52@$YY1UItlwl& zY?^=QVp*B}E~{3yk=DZGT;&UXi_5wlmbZh zR|EsUX=`aw4sFQq<#$?yG@{euK$~?w!rEYX1QgKU{RXk}2wg7E$K|E3r}z!l2wgrn zaS-z>&A)1~1H@_*Fy^zvdz9k0pQ2cPyX$wfi+!=hL>7MVf`|BDQ{3Fys;B3iPw_CE zhUzvYbT`R+*L&({0!_f$G(+z~{%XVISmDU! zxVX%GFVUJRUa!jM(3RM|VI&C;9mbCTDfbg_Ww!aT^m(gE2CwPnbDGbZtDUbV(H2?O zjm~pgFTL4Msw|OB*;=i?&_bN9v&U*=07-L^yu2hjqs%JZ2iBdu5Ka~8q@}5oSFB5{ z@LR~7fQ__)bI)Vh$C7P4;kF{~*3ua}51P|FITo z)oWjN%08pM@;=x*n)*zXswz%4XDTW(cZsJb$05^g6jW>^{Npy`{TYGc4lA0q+r(!3 zDStg2$;KRGkNqum5Ur17dq7(CIRIUM_ZSBZ{9gZ}S$%jZ4m?GQWU@^PLxbAYC1;tM zv)%*)0x<$hl%lbgsgjTu=|W#{2^rF=w87AmJSlXf>Kn~WRC>@*Kz=$L`?QsxTS%k{|c_HCM+ zG61tHlHU?@rbAD&3hSc-zX}+X3j7Lm0B47ffT*b!^ zGNnxcDvyhxf7OwS0a3hI|4`;}@G)yUZdy`J#KQq5RBqJQb{=JijYbm4ly|?caSN zS|+prgMTVFBm+ylSQ4?1&?SL2pQ)?;=TVVkm+LkEc6O}L)}c$k#_S4n;w~w1uZuzv zgs&e?h8vB?I`DcgH)d2`1r830!rs$SQE_^sitYM8ur0X;0DSU$+Vx+piS`8cUY;PD z$?8ashbZMfA~}vXNw$aXGvRMnvz z@u`Csc*+~ACbE2a%)F7Dqd_NElti+!{rYTQ1lg<$xR%C!UunwZ2ZCmpAv zU*|MD#CBShOf2J*kIVC+rA7Cs_2zv^zs6pW<2W8`=!4+Rutu0P{2OytL;a?)1Lrko z@MB{I4j3_R#+FYbDj4g)>2qJW{p&k@LgGBj&dJq^tNP-XLJ~H7wu<-ch{OEgrpLqV zPohTgUhoU9Qr)|D?)^WwG1sKtB<4iS4sGdzfub6*-||Qsj5dB)V+E0xuD=Q>VjLvg zE77XL0yF};q}Z4zxen)FQ&j`Nd)J^H=EYKd$QKg=1GLR1cI^1u9cR`@Cx-%lqKF9hGjzB*v1xDjWr5-awmc1!j6oTKD(&=277~Y+RH( zGPN~<6x}}xU#-4&$b?gVg1_SRxOu@=(FtDf3H!ixtS@x=WoYEfb@=0X-Wki~y}xV6 zsL#E?0srM&yCR{FM=GW%s%Ayv$)Na0BnklA+k_V8N{P`RYA9Paci+n(nU_s%TpnZ; zrpXB&w2bE`nB$7gPj4%K{OHW%+6z4#s=^EUU$<`%U~LJ^^XnIo{AH=!q(po_d@Q*W z89d7wZ-ulZyt09kAUOLd33MYAj>%g{R8a02rb*Ja7H9Lj8pup>RYcojV89C%=VA<< zOqV%`xO()>5RX$3SvddYvnqeP;cBOl1nM3rFHI~V_v6e%%LLvLnoDJdHM{=wW>>dJ zqG}VeB^*A|IaErBhTCVP!-xoPaR;PCt=`7@`eWr+n@NDq+}zy0`%HD-zvfiAzrVL7 z9uqlkF?K{jMMcHVldi(#O1|Ik^kOw+J9;$9%R@+D@Q43oB8+sKrV2h#pbv{>I07s- zNp6JujVdQYzesQBwGphzSTH2fwZPuodC4rdKQGY@x#Ik^TkA^0rd@3*^{RZm;ZXsp zfGe2~E8!mPxOaFrIbx=vl(${e|)fyH2kXB=*B~W{ftqWi4kV-)@2^tU< z&Vf1`!GGti13H;eu|vqPX|;^VyC(9+0Wz!&UO(i0BhhbcxGRWs4Ff-|*WkYFT1L5H z`)CmHgK|KszmXA2rKm*;_d!h`1;13t7JzOAryIgGz>d4QBI&xUw2cd2Qf$;bxmUKP zL3dY0oF?QqqP*O;Z06CjY`dT2^|574d0q7G+L5z{MqcvsbCzr}4r4}fF1aN3duWBGhm4T`5w&9>QVu4hKbP+AulV&jxycUHv^pj|gTZM!Zc_bDWwV8v8GQSge zuR)E(&dpVN2?x7O%{!9nuWE3<{tB)kBZqvx%%1h!}oaW^YM4< z$z%JIolg6Mc_MWh-ur`2k!V%hjDsDk(j0+k@3RbcLHN$;=ZVH^L(&zk@!S*n_H#%h z$aClLYKdM!P;tLMg&_svyg)2XgLR#+DAY1)Y;TAo(m%Z>=;*vyENY+1=@&3TpCZ&}7S3L6RjzoIawO9z}n<_7zhtB`N7I|Cg16qs0@U+t#@9 zh!{QfCP}t+cSgE6dw{d4UxM1%8*RK2gH1BPr2JaH<(!1uq>}lE=Cft9`bwuPOVrJw zHLIfX^0C&lLDr)M^Q>BL>pzdgv|s?t`&9pFTVHta>gu&cEo7@8%Wv^$C|#knI{s?& z2~Py(2v8)HOL;OxM#Al)HuEWB^(w5kUE()hBfJJ=(II}x;#x=@Uoa<&t1EWnZkCWi zqoeU{`p!x3P6U#WzH)Pp&A-<$sv3dEmFNC-oHYPO%7N)PgplC6fo~Cjeu=1eE4R-C zpe!Jc)r`Y{rQ;=LH?{#n9&N%X2WZSS0!7kM(8<1FH{$yWR0ZngGDaC#ViUG-XnVCn z8~4yYD56J#3$e6G8c|`Wy)XHd)-VHzH0w-o%#d+8anZ5S7pZ=>+<`t552#05W+H%E zjM&vbCtBcJlylri^|3!^NJ>5yd7hQH=<05aoj0hBZu zGZsO*k?Ar{PH>3D7R|VaWJJR|8?5=^q`@zk`EPxD#|4H}+H|yqLn&oG=omEF$UsE{ zkAS_`cvn<8aQt{Lp?;squpvFC;Ov zax=xr$*0JxMwW~R@4G0ViZ9Wva2ILVF`8d~$QN+=_s0jt=ExyEqF25c+xa-Clp@LSCpH0MGmDg zBra${_aEDRe{I&^n2|6*0_k(Bh~F{@k$uUiD>GYj@F7#xC$@U$`ZvP`c9u(DlYRxU zKN)W$hcnL@j$+L#@c}<=>H4;ZXiwT2*Op%Y3a_2vRNkW*b^Tg+rCf?b`!02}-=kNk zPk8%Ul}bS!Kj`_fmH1p|$eXrJ8G$8QTUk}v2Uc>4_io%@YD0gY?~YLzJS>d)&HA7f z6Zs)yA)uhDag&zL=~ zgj@xJffj%8QvM`Eiiit@iUB+@A@8*r2wUS-VoL(yAXpR1f+=$ZG=wahr(`)Y4^)4O zx1}Eb7|a)DNG@v(bxtd8N^;VwZ`}EFA7wf`SudB%W~2xx>0D>Wr^SZipbwFT=Zhu? z+1jkUL+O3VX;Yz-tj`&O7^|>C=}ADXfVH$5iMrrrRlwcZT8Chx_=Rc5gHE#%j&u+W z#noTGm8!I>Su=iBa}(r{J^tRh3A>9L+F2=+$n9~NaVV2z>Il$tjGG@8ToydN1O+3I zOiixn{!i@)m&XgL;H(Z%N*(+sU}GBGbUA+g`}Dc*>}nB@f~%$Z$+$?`Zqb+82S9Jw ze70VPP(va0LVjtuqkTJk#sNtY9hwd4kN&4n@c?q`5 z;GV8V1z=}7OBW;%1Nj^+Hn~Z3%W=8B1BovC$SK}Fh5u1S9d5XIxoNDo3V!NvudGXz zAalr26#J7m>R;LP-!+IhfkvzXo~oISmc}UEB_4Z>&ec!tPK!iBaGQ>J&uJ3G&s1^j zWdE-0sMaaWrNGD#{_sQ0{L)op54LhBpP=({l7e(jNX4^WvL71h0&3VNZ>iEqxt|k= z-!0FH{6<@u=R4)C`cgM_0gVeSj5R@iyS*5Sn;MJ~9AZ-$hDJzLHyD-Djbzo#RJRs5 zw9z-WeZ)sElNQKy&8*xw^%ys;^_6k!WWfGs_ahd~#sEu}%l*|~-in#_m!{1penY-V z-ZYnGm#s1PtZjfzxmJm(@^+f)?d8+Wz7^gvQDgD0TI#vdi+ji7;iR>kBKex4+`oBS z6b1dlA!;Y@^QsH(C~s5!J^E}{GU&=<4MrDScdh2s>KrIOR`_o zhbYpQm@q6-WD3$115*FJjm|<{7=Sy@5rRar!sFs@NCU9Cib^LAqYxOseg{W~n&=+( z>@dNkMHj5Ic_IzZunBNL$unWVK6z<@-c*#&AwuaHBib&|@IvfOdBxw;ZluLzQTl=7 z+Yti_5NwG4P6toJ^iN9^q4+xder)_`RITVi2Vujx2*%(0OZYz&X*yj ziWQCW@+RFF<8rH0nMg%GA{KHxEJn}@QyKTO9uXJx$!MZllnSSIE(zdIJa;B}U zIhbd^`TI_dU#Xd7F>LF-%DvLsRnK8Y+TZ3TN{FDVU0<{RVgH)g84EZVcgI0;|7ef3HW&- z>tdUkQ(OS8dlXbbQZJ%yUKX|RRPtc783tXs*lf6=FeVGik>YK~=$?3j69Q7HuC^ZM1E&w;{}+R1G> zai|80sPi*;z~xU|4N70!DdutTGYigp&G?JL>8CiE-#H)4HQx1;6pG~==47QC8yke{ z%lp*^G@N+ub4tPqXx~gyFjAh;Rle%&F3TJxox6QSp_-h0m{cU*)K~j+>4Iu_&b4~b zdJBy$aW(*Nw%>+B5oz{8G+PkGB-7mo4EW$~vp{?CrM4BHXxCo4Z<_fnRXOvi%3hT7 z^OTeE!m5xwhF_L_KJpFvOH3{-$PyzL=?K&2c(<+7)Qv>8&WblJ4qaNZZ?}bW>^^jD{-plX!iFpy5EK+NOZ?Ut-~_DNU(>Ii ziZb5dUK5BkUS#MZ+POhQNa$Ky7e_52?MFn7A&mo$x4e&A=Q_TdiPqG@R!E{*eEuUF zzzv#gC(|j^?(Gj>{Zj>&Gplbw0gjOax96j$hI6v0%b79JO{*JIWr*NHcfS+e+wX7e zoG&G1UcVe>buX2)4^Q$Q&N%YcH3?Dz&w*Zr9>r}oWtug|mt5L03hgOhwr-8$w3&5B zo(|-x)i90tOB0l~nG}F}SMq0k#K7ixs+BTkJjC_JW*Sd{YS?BntVs8v+RID3S;_Vy z^!jzHwxzz)wncu^YONdk%d01CQ*l{O_m4+8F*c5PVs$-`-$|BYH3a{#4}yOz z&3g^UMq5%;R#X^d!;<@RE}9EGgwnXalNLfDpgZZ@SZQzA(Fy&!^4ep;C2#_d_loW1 zK_pL!p2A_mI564n=iN_Su5#j3_7EFnuP(u#7le=5nf9T+Oc0ZRA~`WZ_193fp}03Y z=y22UB!FdaY`h?4tR;dy_h%h4t%Ms31pSS;X zcfb64)xdOjQqzZ zxKZuw@$2wDE)vMV8*q+T`HRZR%B1jSHS+&=2Ltq&%=0?tyxtVM875c>^@wkECCwi z#G}s<->cF=U=s@>yoHUBPMR%2;Brc){kjtFcQ4W-hXbSV^`N@Tn|0)@2pzJ#mcMNM zZ}`8y_Ks`}4iL7^f3PZ!zwmw@#XY!2=OE-D;E5|&gNs@EOk*%sV^0rH@daY`EKsEscNhLCZD`Tcqp?D;C)P5cQHDOKJL_(H*N zz6je=L5^vo=<3b&o?0&Fi

b2#&rT!bgcz?}ZI;gq3rU&=z1)R*@EyAB_)MOUJx+p0O5;|XQ-z8Cx9Zr8VcR}`y&+;CM$vbq zt}QgjRHeKn?t8yqT;=xlqf?nEQM&GpU{$)#g?=K#oK+idDlg% zh&Y@hHpd#(a2z0^_v3}P-Ll6W(si*34*R=F-8(U-y6Cfn)HS9$rN_TL=MfLHCJJ{9n>3NKbT>)Fd2 zQ|(g&P-+XaV2}zwVTxVV$N5cmrjIT?&M!LMU&tAE&i5`QY$|2R|Gf|3T3<~L7o(C* ztx>kU=@2%D^BEITmr2eAEED?OTAs%`n>?U}YWfa)UqBOtyz=|g1jg2@4`bv4qzQ$r z3+<)Nf4l;hj(~zkgZ%^BwxK?*L~s6rc^FUHsb>%Pcy|3&{|gTk&Vh(W+Lc8Hs<;G5 zipG{E=UqvP8T)AhOad7f%jQi?KQMD{IeWKt6aC)esNZrr#9In-ZJGlxr0(_y_R2^Hv$P+-+n=90@Fc zO!3aRbq(oH`jM5vbSSo>%e=6pr$t>l z_Uy6ogzOoId1lqp`Cl(?wt`Xa@Muz*P`Ub%&vQODssLA+oS!2^98Ol=OG>sK8}PS? zJ>U3q*>_*MR~?j}-_=TFlx|(JXUA2+A-@+Xj~<-)wU-ah%rBko2C_PhC88Rx5?5eaSAMbEzd``~Dg4)U>APtm~M zya=3n?+3f?#=kc=dibJ0n=c1mU+)ed+GgCgrMqso2maeW_JD3w)z$lEUs?<(UTGeh znAo#zs`b?sp=_6L$7z1CKGK?qj7t;Rn%Di(eVt>;x89I1ydFNcu2tkc@p$$3X+OlH z_*%Gh6t?1t%U>+*{ww<5r|uj0AXGSNAx(DhSMj4?5CEfsVRO9quTU(=oY)+#6n-zV z&!4-%0T3x34uqIe+qb*(C2QB1e{rZVXbk0GY*im#@dlZeQTE%bR3;)&K?vd62 zWNsUmzEb!K>s<5a6rl;=@Qfcqt@3*v{Vhep@`yU;GwVLl4o)ncYIU0r8V|J$f_GS6 zhHVzCJ5{RY#=*KVLigGyopT_#Dn{*{w9aAhGB++?MH)%8JTAJ=pI3hVF|TG@b1&BNNSSb;qa_J7 za@Tb)UVT9N{rfsJwQUns=N!rjEnLo%ALh`(ZfyF=V8+6$wB*+sTD5!60C_P+-19Nh z@?c#5^5GVtCjE;FN>{;K9?#MKP30HrU+iAaEA^K&KO$P}49)E8k#}UGszIGYBa_PP z0~@PvaioQ_G&SIVgD5z^i*9A=G*`=Lx=7OpgzZwqrEo@(kfJ?INOQ@vmL{|BuXVEz zSr^rIKX|5Nyc`#a@qJTSf0I?}J<3Xzw8INR^J0Cvs+&nUJHOx9lM($OYC+w6?2-pr z6OTz&V?r%2?;+Z}4Yz7^i6p0%tL1OzJsN7^qsG<|xJC?ccY2=hkHx5F+OWb2M0KlC zxI_e)@wkLgDtT#$(PZ}Iw@eKMOLL-%qgQuuKw^A0|4vcvMQx`E=} ztl5TZdnC)@|D_=Ft*p0fBsj)CEk}{ISreOe3M9^wL!Loj? zD+JJ!843Dr4ra;IS703jrf*3T$adJ3SOWd7Byeu-6zp%tVH^Y`+y=>pT7QObc9kBq z1RD(41Q{4hev=RPhzG1{vT=}h6GJ1h)fV{ehAvrf-;8k`pv@uw?7U4&PvUSz^PPP0 z_K#gj#I{MCIw~?XLWQV9C&5F`tEdO{+2%7bb!%IcPged2<#iqco`FE?a_#W|LpJ3L z6Iw{Z)^v(en07-zM8l8(c%wFTUzA^z%MgBA*`*~i3GO;`#Nh3I#*A=sfK?1bjQ!VXa zhT2Vs>CupdDRc}3viAIOvdh`zRv@QI{YkCft;Oq`sC%AA z-;1rnwRLb@-$q{B9_ts6&`^}91fc&P}dBa?{pAK4@O~=U2aRP9w z=a{1Ll0GKYf)-RUw6^^^?(PpXxF@}za_+UxtM_3@O)CUo?kstdPC$0$pdBYbhawu`JB_0|S%@k#)8pGh zAMjWHqSPqw^X4qP6_(n*0X`9CoJM}W{>baf!7t)8X0ESr z8F(kESN^J$1^9cE=;|`@ymY496s5f$6)$%f2poZpw1WnvnC=`c6bB->R5i<$v1bSE z;`wVI78LFCNoALJNCo#P^&PP1u4(O=KS_$idGNEc1f_}&_iqYc*4O)k;m3!Q^dwJ0 zC;%kTagqDO_vHFj$GWHJrUxKwKWC<-8GQNMT&bg8{+dIQH0E!6^ReHfgQLJvtQhVa zIoLZmbT9Uy)|0wfTerJrwdSX|s4uIL$JODjotLa2mqA|o@%;4precj(`UD*-1%W-3 zt+{xjUQY}WfT2-?))AddlQ@|6KHp3}c9EcxG_NmpZSqyT?ceP)Z`GI&Sh8QY>NzZV z`pH6YPE=6Sp0;e;O$bq7D=1DbWSQHXN)XwbN?2aK%6K;6Pqc8Ba`#$!i^U(Z^Gn%> zRMu|;4{~y@M=qN$Eow357)EL98O(jQ-YrjiSY7{l*Z;V^>tz?z(yxX^dNH$@>W5xf zODOwb=vH;B-&y-`t=eB-mcrw*D@Nz7xNf#`d-?N3^Mz!i!qbSS9Z*>Z@xxhyj>Biq z-K{J32Q3x-{w7C5!wyOT8@OQt`dXjoMfJQ@#lpmdA<#ng_ut8>uG~T87_|aT%XZB5 zx(`@@BQ``jOW7R40k!cGRpkHAkXnSeN&l_Y&=a|65%^B2tGR6TERxsXnF`W&lg%6X zhT086M&v&l(5loA2HT8X>ud$s4ScFTEH-<9a9im&%Vh4$YLyZ%P#gU{Witin$)MX# zU6K0_QIH|feod}J$~=up6$D`!yMt-NCO+uPoZ;wmT_kz<)2NsI)T8K<)C{+sFXTit z0x8#tzq+j+fTp9kgPbuSE_@`?W z_acR9Mzv>JYAS`hjV7j={ItRz=iSvDyiy)yVYJ|-Rj@gGy`I^K_MOzZ-gYG5*v~eW zwC33FifE3pO8B1MLOjji=+F>hYWOGS+V~UEP<6IlOcicZ)HRmF;j8x@6ic!>ZC5??;`)SGSOC%#z(A2n)N~Uzp`Q>&n9H z%)S-$AS|1HiJO{2zfB(_qf$PGfbvv)F2yN4$9>Q_uoi%qMmHwqKev<*aYFvR-i)iZ zsOA)Sd(Y%XPr&J*mL2&WIOg29TNB2pwDJ_{*Z*|!9DjUNYY=&R zR5T!Ty`g!Ov+cH+l)yju;&Nuy3MG9XXAHj_{rV0;m#in&acOP3+bjGxn*den7a^h} zvxl=4-JUn?)J~s;Pk)s8w?8@^O}{xP`hcZ{iO0s;pJ9MxJ*QZOX@ZCNYY;z9)xY`B zq{d14UOPzXo2w5pa$Y{mtkn=*D|5^`9qJLSD;`o5FKZSU>n`BjhlsQsZwGnXc&N54 z2_Nap`X^4k&KQ6EP{vXZaTalPxPCT4Ju*674S+@~*R-b;ZAx`X56S_YD9<*UDqQ)1 zQALoD&US_aR-yxR$vyqtrxeYiAv#Vv7oweFg>gV^->dk;u*0YHyur;TA8bKSW{;4bS|5Be$aWefwnZr;eFq*&B5&`SbgE6EjZ*h%F9 zY-;Zts+&|&wFDdJhXE*tpuCW0egdrsL45hij3<=_Kh{9G%pXlH3r1-ke*-mt^{HP_ zbu_`uz~TdVr{MiGyE{nkv-v^nj5dJB{v-da+9*vY;+`+cM1h`x`FT}EIM6_JgXQrr zamSGb{T#=qnI2D`ykd<_O-*e&J3CR&a@|WqI`vB$8s@3GfZv!yvyl*~B^pN$kA*oY zA>z!+l9Hi5x}+5&0-~w&4CV|g>|VpM7r3|$t*!6-e;O=QRaK$3fjV*-hqzU{Ae8Zm zG5!7h+8;l5e)NQr-@*9?;qs=RX*9|+l3}y#eU!hsRjka&RCXLKkCUm#%LG@+o<={J67l#TKIpJ7dmMADX7oNr z@A-oV4IY=5=V#yA;hwc-gDc3Qi>yUfV+u!@5<42Mj)5TkVcLKtSo0=B?p-FB3l~6BU zM@RlCDBz^tKvYj!JwH7<{bd}2$Fpi`efrefEX@WilC9sfUq9hR5gh`&jO&eH&EN{RB(sQN&FqsjDNk1mp z1{@t4NB8az4WW8WIbh~60lDMHg{you?N??5+b7lwk`9E#-9;8@nVBb}n*t zeMti*^7A4J)GmlglBjJdaG_UP1h4^W+{O|Jt&BW(VACKIKM_%zKwcszXdqqTj~Ccy zB+#b~H5#-NN!5qD_r+~ZlNuyD{WJ4^!qN(Pjh*712WoFc4cKf}A;@H)v5}Gr6!N$! zWo=(Tz?<9=jj8f~iC4zC{go_7=H|ZRLOCpf%B})gI`1GZ78V158HC_f$G_y#Uc!XX zKr0y1VL1t)A4>P~i%bIZx98i7vlS+by|GNii-A;bJ-$_=#%~dTP5&^j5V}%S+ zOBxQ1=J^`yns->+7*Q{5!Yr4(?k%qo&dFK?R0nR+?)T;}udv-^$^CPGDvGcD;X=J* z;p`hgTfwr<%s<3Stp=8jyr#lC`|@GLEc|)N(Rs0E35dOX4iDJ;)^ud#qK>uWUVFaG zI)z*`Upz@YB~x$ZTz<{q{O6gWIHv!lflEf2?|h&rUcpLbsIzQ)O;8?xONhT8i{-wIM4zktjtb_fGMo=o?%~I`t$t{WEPosIukVJvpU#geG9HV{eCs~{ z}@NF8>rJ`oX!xbDmil&v?XB8{~JJ5 z86gG`%bCXnP~K_6`mhDLGPRnNpGaSQrLse@=zv{d2-8Zg$@T9+SyZ`?iass7kZulEdz!K;WD5ec>srD-{3e)vXbGgKqq@^VtBxfKr zNmfZIcD}*+FK0?!0NC&cX9Ty&P%WA*0waz5Wt-Cqa4tRq!85aSJZoN zkTRx-@=Ct_CI_b)Sqo8Tu{9Ir-C&%9;9 zxDDs+e;axuvRUxa7xGuE=|nK$;NakK&%br^d!nmD86*|zRI%9YQ;Q*_V>6o(myI;L zfjV@u*x^;7P);`8hTh`8P3^qWcfYg^1jTM5_DMX4H?L(C6}F=z;Ti1xy4%&?ym~I* zT6Np}9!}eFRTp&(zHJm{!DhVlfKeZBp~aFD)rzn_3-G=s2Wr-tieRr|0kf(POXOLd z`VZ}?x0^#+TALgZM42NKuPu{<^gfZq^NfcLrbE$31fI9>$ zV^!G_NKXd^#a=H=*8b@xL;Yt0wz2$Y1~NC8maWEolf;xgL(x`<(TsR!d#;DPN0`R;^EpA!dT!!!~j!8ut}FLA(u{RE}j;9O9c zp|35?El2*@4nEQca`*=W8LzM~(m3Ne&h2oxLrg5p6L>;|l4CHd%_BNDNGJq?l8gv& zidtlH@JaVHR~p3-+g;dIu$R_1T_7*}zbogQX+G3*?aIlhSWgk5;A;gDtJ7%Q?H(MP zF(tz>aa7r>hl}GM+KIsA?5%(i%qR?-2DLTeO@N5&4(MX@7pwMf+_~S_fWl>y{yG-< ziR*PqOkYk{l_UK_#XwiZbk=o~h26e;+i^5||L6nD_Bsd0#8CFn_Xqc!+PlKyO;8_M z^_9m*%cCml^zAKK4 zt?Z>nPPWmlM!RO#7Wo3_uPw~m4+7hL*@AgoB#ceP;}=r5gwmnfe(~cMHXZ|yCqmt< zf34p-o4z>&`2~m~ zO||cbzw8aNVXIObS$sn><-Cc@sdPAsz* zD_{Di3BTbR{vQj#;9C7?wC9^G`zAw4Lgg8YgIZa;tg#BuDJbFoV)DT<`-=L?52Yjy z<*8pOQl~Iiyy0JnhTjJTMAL}bOka1jwEMZB&z{NRWv6xBYWd$G~5{7iusk72`tRWJfE0yQHtbD zY70IA30}n<`1w?!TxwWVe|2lrkfd$m0(_g(_Xs1-i@h3QHa@t5{5B9sbk(0ctFy>e zSSP|#UF9U9E?UO$X($aC6*I=l3|+_Uuc2?50v%7j;@52Gu^6FNa8%*aJVY;=x&bs2 z%JkiJpC&sP=*u|EerD~5*jZ=*I9T&YIVid~X&TLtOz6yBOO|bY`C7Y_`rY^N5}P>b ziHqh8awgoKkQRI+%mCFIqa0HW&>PUg5H;Z((!A`{Bx>%H2`5UKYnNRjnK z)3wU{rV7@kbtcLd;LbJE(XCVvl*}&L9eN%oU2Kw*RrqlEeA_(leqAqZt+vAPG+iYP zYq_zF;&>TLi(^U+bx{;G>xX-Ft_bXv8=nn`g|qCO?Krd)vMd}rO$Z-VPkN|%(bHIF z;}2t&%g5bonWfWry>hr!TgKHwggByK&uKq+6boHdNyVpr*xq+vgv$CCHXmE6E~I|D zkE|P1##N`l?o!t+@AuF{$1nZ)`Lf00I=X(1a;xDgEVX>nG9+X*X58=S%XQ5Loe5SS z(W%5bm_^G(B5LBj$BCbZ>hc5KZQtX+@PpwN!fbIMdgGN&Zk5LVwQ9XuGvzhKW12Lk zHQx=(Q*cX7xo@yL+x7N@fe&fw-8>Ou4gZgNrIPi(l}nh4-RaE_BaGto&h4PUqJY5H zdNA<2iJ-Pqr^qWG=ZOM1Rt$fEvhizQur8PFzUCK=t>i1GPm79u&7l_Y7FG^ygZ0T& zV(kMVuPn6;$!%yWkkJb{Ula8(paYce_xr9AD?!++c4oL)=j5Nb@=;))%cueFSay1I z^7FM>qQ7k`hHq)h{F4&mzNV$$F!OWJn#^7f01JG-6^l3wpu?usyyg$;EXE&DE^=GY z(TVblk?%o7Tu_1jgEf7cb)IXpTM7slg!lXnYksT~m&JzL;q<;*j!)-6OJnI*@PSOP zl*ASFG}3#0E=H5)Nwt{^zif%c0VkxNz&zI=n2$x66=&9`nJ$aBD$a-=-Ddu0Z4Dp9 zHio7Xgh+moGiS6N=@SK&#D=D6FP+Ajt}BXYhl-k>YgD=y)crU|7(dN{dbF<>4T%6f z<;Rx(kj1#=MH5C8Q-&mq#+UjI#X4& zmNaC_ZP&3iCbf8#94SrilJc%0$H~Pv!2_MqFLRqjhZ-n$SXlNdl^$ z=cB~EoAVR-{$lsDLm#i|BD=R^T31$rk49DC0ohC3jxE>@Cr_@Efmpey&8+%t9=U(l z&+Rkrw_0tTJVTVZwvE5m*yubh9@l={r-%t|G{n+NG~G`xJC~gPdfX}`vfgnh{(InX zI+a|@422!|Rh^ApE!5dNVLP4BMGEV01ERz+|I>T^X+CPj zMCi^1%BO>X`?ceU?ewMfBy zNN@EB>~YWt>d-;Mzj)iv+zQBWAAc*iSJ6MyJnOFl>9pk!!{H~$8Sky)B{9OiYIOs_${(isX!%(kW$S2MmuTJ_`U$Dt(*@(&pKhbbt>l6>eRT#;ZTU~0S_-jN}pgJE8MY*i;q)(;5&KCmLasRIkruhUbI7DiUgQM z%8rYRA2rBN_0M1cw?jOTYDetK&y88fkS}IPo=AupWgIb(mC>Wc-f85Y4k*9N{Sx+Nh4rLRmb7! z{@FVE{&tzd$r6crMuy*G3cs@&-tB+Fkpu5jSMWg?kZe&>=v?dzc*3I-X`~lU)FLlJ zbmLkIRtDVo^4S*&EsuxnC^PzPhQ;h{bTTaex+eXxJde1?&3JmCYH{zkK5o9Z3MGaB zX8(F@vM|1QZ=<1d@?Rf>`+jka93MzTZ-WkTWd65c{#!JIL~3NMnuBz>1}P>!L94Gz z73)pFiz(?JmH-%`Ku^4%-jUJBoxrP5ynD5l20{P zt4N~w@d;_1H7~SA%kn478>%5TC|iU(f>&aI$~_iTK>o>6qIJdQ49&p{o=7Oz4$dO@ zBD+#=0bJ6@BHmObm8I3SY*UaV#s(zPc+u(WM$)5M2#b?(4=hWIxvCWz&Z+~19zjz` zURiY%yKMZ)qTsth)OWJ!ZQM=Up24dKL6Fr72>AT8rDlv#7*IUDr8&UUENtX?n*$r4 z5PcAx&)EY{)rcLNopRbpL=LZ_0 z4Jj?at#!VT+J3*!dOJbImA-1k_JlThH2 zgV-&#e2i%=P49>u?XJ#p%kt(_8X@T~d1z@6&YA6NsG-sJo2@{VMhrka;}3EYF_zg1 z4^^ew*~5}&T%wf{>x-~0n<;6m$6))pt{3@T){g_kSG5RK76O>Hnf=+@YM6BxN1;jy z5qv-bh}y3GK*YkxUA%)w|fk zj3E0|yL9b8gP0mnhH%gw27i8&j5B{TPq@og_)yzN<|6E&FxHJvgRSJZ;#%sUuE75` z00nz>mlHJ23jj;tVfF-#VT>mo#5o-cq1T^+_ zncE+t9GGdQkn)>7<6YR4pVmfm#Mb67p{zNk*NVLfCbG9VdoB$(@A{cDzLR%-(2nqb z1-2;$pNulq=oaQ$Cpm4yPh9~nWGnMX)8~DwzjIiG+`*Q!Je%8q6P|64mWwPqK(6=g zvX0K^u}OD&y&#{>SKCUE1%nWbWQ0&`s~%DKv<$XMk)_5@yt9pc(eB@V{gAmX==^7! zFq&~_eow$~OtPb~3Q~IG)-R11gvy8GQrSm}oZ#;QkAEIxstK>xue)pAdb@UtJgZu= zjV=kOKEToPv>%ZI<*vd3_M2mXl96g`*NW8dNG_)a%hu1T(|j%vi|ramuPlc^>MwZp zj5T#t_|m$Ht16m~2$SF<}E1oMwe#P3B{0ssOeX8oG6WrXZQP9P~1uGsUEm$NB@{ zDvX@xQ5S9j*ho}s?a?sI_x7Ftg|_geS`)m)n!DX!3*GO+)YNwUn&s2eqlk{f4^M=+ z5Fsnb=@=rUhO)rjX9dakM*&sX)^^4dFBs;F<*8muGNH20m%sNJfN_Lj1Y-?fq1X(4 zwW(i7^>D{aNt6gN&~i$TM?Hm=@kCW+>L`ygZZcxo5c^0cpoZU|{(A%N69J0AY-~;v zyGzxUDRctlnpDl`L^UL88p_H>xF*PQEMPECuEN)^Up4GUakio73%1>q^;ilxrz#(UMqSd`5S^ zv0bKb<+|(dJumzhx?T-7PoU}QDU1FdV!{(|rCPl@=!yWBPdnA*dBr|-%A6Yjqk@S) z08H2PSFYdEr=w)hm2}O{q+s%02fSL~TF5=~v})R*TU)0Dg)Vd72I)t?!eof<3K1ML zX)c{Uv;Qo0hF==+X&xClYmx#6<(8VG2DBhd<69|!hqXeRFn0mD)nC7U1<}M*I|+~= zhB^PcO5OgAbSBTWwXUQ)206Y|srrn&Ng<>6*HNZK+ov?>;B52xG)#8M<*XvSevIxM zKM%&nDiZgpL=nk7(uRm8&!myCf;bFD1vw{H`cN z<;czp&L^3|3Y>ZSg=7}p!Ns=Jq@yS;YM`?gK7P9UBb)Nw;4VrfmC_0B41OF)Iodvd zlES=_!IZ+lAhqjk@X*6fi+qYVfX}u%y-2r`zy-p*Lb}nF53r4-KCR((Z4Zd zGHU0t-%f(`@w8B`d5nq_HcS2(zAh+`h$HSYE4Q!us6%>U>!T2&~uOWJ0C{xH8J zQ2X8h^5wT|R+GJ!nQ!{mWY)_b)}M!bGpGuHZcx%bMd|8)a57}v{*cDp zHt}5QPdfI12ShPMq;pxZ&67pY1ghIHqWSA3 zaT@HAtdlL&?I-<8{y;W^i&6tVuWdE4>@k2s7!NdpAx)}Hk`P%5Hd&7{+ zZ2s!XDK>T`o&0}h+W!o--~aLFRPXtJ8hiBcWfEeWVkHKbumOi}22#8YG5>_7Y_3cd zXwhGDB<^saIe!??viC(X2PK2P7;k)ct2EbmCt@Lqw|#|dEwh5D_)0K7v||pg5~q=@n(YMW{{$F^Eeg`<$KC?Clx`|arS>nZ%dzklpk1_ z_HqP?0NVzumbn!hRX(8}Y<1Gny^x;na4AJR=E0B6J5eMxNhQB7_-@@&HW~=`HTo`j za-j9vdNqw`2MfZx^$NN!6J)Wv`+jfl6Z|~7aE&*$zR%fe`gzDMex4Z7hjMJ{@$$zX z#fhnJEScRCPm9>KQ6}HSduRg}&w}}t5A8)PnY7#=V7EP3z|{%S={?cvtE=WYLC}Mf zqhlQ+nL7W6{HXf|Zy_TkIhg`>f)Kj2g{f)`7Uusy$IHn(1g)J7zQPI2<@k#B@(>jr zH<(hcG&pAA9af0XI_6DqEC!_aj0+|QQ}s_*+Q%nBFg9UR7G=}Y%W8nB=i!xeoy9J@b^MW@W*oMDL7UW7))avOvTi20o#y&azivMbTViKL&*$qKci3hpN1|Sm(H{x zr}$k)v*R^u5x)50IQ8fw1t{E zD<0uS?pjR_VS3C+Coxf>h=#Qxfh*D|KJMsWQjBiA%T_)IX{Z(r*sO~w#Ri1!kqETY z3`n>L=mau618lz601P_N6z~&5E%scg`iR^C=axu~W5f{Da-5%^E!%6F7FaJy#WcIb ze=l~$A`C^~CL5>5&UJ-$((4aTtN4oh(}&N8 zC1hLJ8}#!LS^yzYExo|I`Kv`-i=Y`e$k*DTjkqW#BQ-|bU)bddbm{n5<>>UCO@{f0co_Kow^;vh+JmqYf=dEtn$5W!ni7Q<#_byxZ#D7 zbN@2fzRF){6Zg7KrB{k4e$gt-G#%?hy$2S}?8NBks(x9b8H&C4;lTk<0lE&hea5D} zre@0Ga^0}5<=WD5Y)i)d@c+>Z{>dG@M71*&waHT-7Wegm7ZyMcTQ$9>D=aM(8aCE^ z)rnP9!OH!x;E*A;0QeX9*Hr2LBp7ff1A`vMo9m5fL?88f-rlO8D&l`1+o?WK^5 zowVG>sR}R;3G_n)hAamG` zRMTJJT)+jZtQ_>q^j2Hvb}j1=CKdng+Lv3{&e`WaGE?D^u;*t##|fsedH(UULpFdlSx1>eos?98<(mWO@$tL>6C?k>-wp!}JN=2k!r|zojJ4 z+?*kD759T%KI?)W+-@9t}IDbQ=io{HCGj-i{L%Jik zQWWDxt5K*#)YnzMpG)gUYS-Lby1Hy618TSA4;R~3>V9RoGV4oId$j7Nq-+h{ zrlm!8A|EK!5}^bO@n0nO+sAbsEPk_%L2r)X$3AChHmiB zwQIhK>|vJo6EbD>Q|{h2cdosOx8iC(){rk^qqWDCSkrTHQSm)`%On@2qQYUhSnqh0 z!)9q8xy8V)@w{_SpcUbMHOY9izPPx^_`bDizF$epvH?-`@NY}@?*b61HKfBe9=w3a z7r1;j7TJ?kxKy0CKzd>*>bF;xcm|F&f(!_5eBeAPE?GZt>afFbEE}_rFtJ$UJ;MkZ zRo`Arb1|qPxal3wQZC#Q^NKNwM0-J|N=kAqQy@=8ZuTqEkM&?3Bw2MU6W|yDh6hu;SdZOFrR{OL7w!$z_ zxyU+UtzB(joC(IWx(0^g3?<3{CM09g;^|&nsp8M!*s@3_wlYo#=h9jW)2bW-h+qn& zx^uaqJ1>>??Zg74;9zeFI|j)pxZ+N@VFWoi3X`Nt(}P{Yr;$)ebg*9;3;(q}^4QIJ=k^LDSHrbnY((lHK87P4yseO=(<;y&=n(pHO#3+up3efxle%yp>$zcfv ztfssM223G@@rHOv=VLI@ZJjBGthGqZj9OLd-zbT>$sQJuEiCs%TNO@^X8_M;lY_To))f!vz!^WeE6Ob zg9;I1$>@IG$R>fgH;zj0$lf73T<{vMB9Mmgj+I=b(-=%J%>&sm-Edqc(N7HtS2{xWc5gD6MXO0<=m z*mWVe6i|xgy?dY52v<+-OT|O7P6BMyro8x3C{)YXsF^nW+WuQ!{(fQ*c6GRbW0iD9 zgpV3J;~L$YqxYu`QCk@H``U$FsuHwxv`ij57Tx6Cx5R>8!fkgMX_@<<>Pv`=zB5JK zD^O)soo$!ESUBDh<`)<9qT$wr#E_#mid>9{8@2kPTy++kBco;@9Fn?CZdHpqeF?(N zwTgxj?9o0I0$-3y`$hZu`;89gs{R$SsnK!w)v;| zANxK@t*_5+v&X%_RR0~?fBImsy_^%#nA3JvwiFZ`w2e*0-Yqt@ygX{>kvgPxK3*Z& zZzI&*drQj8tr$b3MuFt)>(3hz&QWzi|0p*_QZ09MGKO`BWSQ~2Wp`{=Twr%Z5wFBd?V@6jwou# zDUv&uneZ+;Z7#gnesfFSSJ~o)YM>hy+OpiCXmjc^RivKgZ+g#dtUFC;gQ5jgg%j?4 z0;08vFmtzUAt?uo0sg3%9lS42uT{*(K1s@}YZY*0=97BQ7lLr(`nR)&@0Aah(%N*+ z+E1JciGQ}sI}maO$G64r5(R!y$zTe;WOiF4nfh(LiTUx8UQxR~T;p5BfF8^|{he{j zs~EeO@ys%u&C3*CXMsVj>IJvJ@oIB}g^rIa6P;y^HBM>uR8C|TmeWR%BV-Y_D<5}0 z^|Y?ru;JX^cSE0?>>Bc{(&(pizKc1IWh<;sU<7=DB6WdQQCXKGo+QzC8_PN`lbH(BQ zi)`Fc!uN3v9tFzTFh5^U!0nBE(@=ewBcSgPvz(NLMkr>xXz0C&$n_RxEhtMD2xY}d zk7H5oEBxF1y(S;zV=|O#)^c6z>dPMK;l=eeYHKq=H=dA1rKcm9z^Lyh^VX{^r*RjX zn)>>=tJ4Xk&%FpKk@jE9_hIPk`_<~1l6jjHFytnfDV>9f>#r@Ze7dl!96;}B(qZr= z&J)8&@v$=W09M)$Si2wFPgZ7#!EB_&9Kr%R-V|O4pr@1q--%DY&wWC0*plC@4W@m7 zOlWgr=P@{ZxPoTO|6I*3Vm!uVf6=PKP+>IK_+&=AL8DrwncjVWx<7sdtGw<`S22oD zoN*gGqfxjNh|Sdh#=$^`D<){3w4ja6x}LymRNxAV0oWwK#Zc~x;I1^SER zDp@NI^-kV!qxdwHL|*xq<1vGag<$b@>OX@yFn4}c`ySZnR`O)5+GhH_hOh643Q|8@ zWr5gzpYKgRo6?s#Q!D5i!8;AQ&O?y1O}ufI{E&7L+tw_R`g;l#rId)jw3VO!wC}lD z&1TJ-Fwc3@XM~1~3J9Wmulx|Qv->Qn5#qaFym$3xq;TJTLi;P@3;8(IFm6SALP#Dt zfN{B2M^6Z4nT)rdHajYLy_*!9{#bgXh@aIe;b%4umgnFd1%ZNoYI|y#K{?JYH1X?{?N4G$TneWL*o`x@0(unVr; zV{iTMVSV;WAZqya5PNVql4o3I#fDd<@cFU`jjf z2yw0HC~~wQDw@j+7i3f>mmLtRWNo^xz9nfU*QsJHHm^83DrhZw*im7zN% zM7l#5kWv~Hq=%4>p+RaW=@O+&x+D!-ueEt9 z-f{RO=4pV6pKE3}1d;66TlR{&47=fq^v)7ivSX$T`Z#%ic=nR6ND7JC7v9WWWj z`36*|YqxNle5QhS%zPQZXfLbvPC#{nc!nBS{%NoJk{ChHMx;YmQMKldNKN})Qa+G& zX=MgLC#FElE(oKlSRQPQbd_*nX3g#C;9MV`BVN4>)E+^fdYGY54#pfcOCOI<4l~DC z+$vJv=AIx#xKK?$kJb`HcOX%Xw8^_IwfX&m=6yL<6xzD-2&g#xN47D9&+M%>AU!?U zMeo*D3!*G7G859}DdOuSkYJo%(4Ud|#kxOwJ=qf!v7+&}GDidVMo?I>MEroZk_7`w&_%;Fe^r^hww#u(l|1RNJU_jEg^h!)o~+>T+_1&-hi~WFr4oNQ-rV1j;m+XTaAGBvuy?EFYLqhmSeOjg0{0oqNJnY15$nS%F5t)&E@|x;$uka ze1{52&=N1WZ^}_L_buGmZ6BSV$@nxSv|BsF1jNDKqpjC40eBgY6;1k0*d-k^I z2OH27fAU$4AOXXIDrVspcRah75}t#6^&}sDHs8C&k|#9rc!Km!74kCb>YReSNHM@_F62Xo%cnAdAkqkHr&nK#(o9Sb(SthWKt%z~+B~YADt-lGOqzmJDlS22 z106&h!G$S(J0bUIMYwh!yE1UASR5&~D?b7w15+^#CRDVGf^3GKG||N6w;HnB!^Ha`AJn)Ni{t1Gsb=_Q&sSc-dxT-*u;-Gp30Y#I@0x^I`;f%#>-)ITg3Y znv-D!i^gO7)zCc#1R@Ri%4>sagmJi-=Rw^Z-q&a0L}1JHs~T3o@3a^+%$U!4aQ!t3 z7?DbV^_&N>t5G{hW z+MKz2wM_;Z6TC2z>nIrOu_!r)dii}Z=;@`-qtv2~nUx;3GdtHe?(4TM0Dw zEXsxcF|8GXA2WnfoA&+!bjfm4``?}9Zo$GL^lB`keYZOq_933rEpC$P|K&6JXmo<$ z&_Cr}_DXn|Q1GwFG(kRVzj8@dE#VKU)EKNzi?pAW$j6`HVBsKtsf;H`Z!#y^D{*m# z{DNUAMD&l;-0{L4olW(I!wmrtV%-AOxu_Tem&x7nb65Vsbj*?6*f(Q2b;c(>1AyF=l>-1xa3nChAyUSmZE@#Ngnmj-TWu<1!d#^b%KF=zZJ% zlHLePu8Ru|`~YeXx1O#*3eE#~D?(gDO5Q23tRyL7#fZK+8`C4fn=7>04{BnCcOw!8 z#f0d~DknHZECdMa=Ki=B>#4s+tCE5Bp$%h26r#F0UJjLvpSskb$ zpe;T4OmZfab*JSC*4OYf0B9W#3#-BAX{_=AVLvbHuMVW^l3I_Ugz`UtZ$m`~jurqq zco6S{_Pfou+4sp>;@G0Wy|1#GFx#-LU+nDyI(?KDDG{D$pHnxJG3h*a$P!j5@Rq9V zp(U%09RRl29pt25QWBuXDj8i#TBm?-?zjOpDmm`;W_>^y_8BPu{pyw>meHL>cKk6xo$~sh&>O!VP5VxA5q4D7s2Xn00oq zVzK6PhMrAB84o^jl@s<1lUUL+MzTIMQwnVDp@Lb_dqF0qZs)Ur;=7TR7O%4+u|xol z3pR0o2hi^EH6T&7V?_t_1rp-PGqbN?43SMQ2WVmxP)e(pI?O44w$JbUkY4o^M(sq1 z!LXP10*}6d0~`*2x<;M)oD-N`EZU@JULj&>w6VeClG>!LG#vhu#1Cq9N18(T4$F_l zf;Cl%UhsK$wpE>sVF%Z@aHp3I4W+MjmVQVSl2gS$6@jtuVz~pQ{dMr$?U9PSIZMN!i9vmYWbLwXP{ zXgjis>O18!t6XyD@KGyVa+bqRgPx|jykOV+;*)1jKn3}OprQDo#4K8N^Q0`%dDo5l z7O)%B`r0N!L>1(?QM}-(W?yg_YTInIQrW5(R`FJoic#(jV0D8a7aMp_F0R{4Jbj{! zCm*lc;Q-}v!24e)>EMDL46SY|5joIS`(8imVFJE~f#}$XuqM&{0y>oUgYZ`ABa@m5 zuyQZ3v92taI>{G2|Kv(R$yHE$9K-Ob`=q}Y0>Cx2PQ=m+#{05>PeB)628G8-rkc{Y zmlQ3BL%Bi7wxW$a1p2xHTBOQUdvtoKX=;b4vT{#w)chymMQwg_qIr3Z-TL~a%hnt@ zvk?)pey~Qer2YsZ7eQ)MZ0TMEDq%+cA+=sv1Q%&j5H28B$Y(?M;;kS<%uD*a9NZQ`r3?@}zFpG0qd_%w zt2C&F8QyW;CPm~6lSj5TpM?>Im3P6oon89zrBA zJ<3PeB?jAst5h<$ymRj;=8f(n?8Iww3Q>nv9|9^ea#CJHxDuZ7ZV^wYUVm%zni2b%s-N0*| z#U+*H&!BYT6K7#LDtq*bp%)AaOL@q8VnR9uDzbI2|h*J!FG@6+p4 zV0uM%Lh0?(@r8duJpn_53?tRf7lJVuP&c<+Qdcc9yB@a%lXdg^ut-Q{6Nl@jNB|97 zFIyN=$0WVfD@KP6E0CF#;G|D#vYUP9>4z3(VJ1lC7)EGwpC5v;~F!PCOb z$JrM(D{m)#_&M*eg5QIw*(sC8%Wm>ZZ^;T}+ZtnV1%i@Wj15Li4jN=u#T)x}l#FG? zH|kXDB^gt#Lp3glzUw+De~9KT(_=!YNrZ)D|DuK@6Q44-9>ugkk=&!W zeDQ-H`dUpi$TfU#wNM8Z#zxffuI(I&#jZisbD&U46qgsRmTocp^ zG3iyPf9HBY8{s)&S)Y@#1;Ye5>f8FxA6)8%D#>{AMS`C$ZHPYG0NR)R2R-(=(x|qF z!^n)Pujd9lWTuSB`E}%J3wg%^Z~}xn?aFsA#}|aZM#f*p{Pi>ln8*kCWANxA$uo;Y zyUrri;#=$X%+2Pfs&gMmnqNDpc82p+?tnRVhHt)wi{nmH{-HZ931@g%1`HYYb*wXpAgHR!hG%iFQeqX zL*X~C@B3E2%=}|R*U-EAEEc5m){Q*$fWT$u%hI=uoRIn2Y>H!|P&>2amy0$Za%Fy1 znX09CQ#NIH2x%Dk=r92ifdo9X z%cmLLQ(elgn`xdzF+E#>++oPE!!yCP7`5(fh(3GuLJe|RsoFn3A{}1007W~Ar)kp5 zz1oX+HR*Gc0_{cMPAft?yt(lKEA?XH4jc#tb6~RK(ZMHUXL5wMDoEsYU> z?d&RX{b%Z<&4Xq}A{(=6ooDzBETm%5;wep!F(A3|8v$$tVQuL4T1h~T49?R(pM6hs zA}v(CDP{XzN=V>d&;cDj+Ve!T!dCSsKB@P+VC#J;gMei)a>yFFzZpN*DFJXW)~}-3 z!(PC?pCCNL`VDVFr2J_pDjLWZ{5BHM8S%>p|qRA{HsrhtNlEt@+0(^Tu9tZwCdf-mLU zW4pWKNAwICsaRrBCQF&^v9erWzkVI~j8C=EiA@JK=(E3w-kL@=%DLXPLQ|y_4)4( zPzyWJ$Coqs;+)2a@^5&nq|x#n`37ajZaEtkR(%ysX@h)NwdgwhGaBgytQmD6Eh_yI zyNIapfzPDCw)d0?@#YTD7qkgl)S7>n$ScA{f@ek|U#BE6-LJr3bbP~Geuv6+Spjv_ z0=k_SocSg1ivvuZ!-!q|K)^FsWpECLGrwf|C<*?8-9K~GhP1EqP}>1{N^OB2(QF|Go(09&c=vtEdf!sQ}QAN_5NB1vJoh z9o+uR>f&MSA=gwia0cH{^McZQ@VDNew$sUPpsP<61PcIj^@del;Dh98=?y&{JVakl zs{9$LZC-1Ewbc^ehE$>;G{uA_ST`rb$n8?8p1|&b)xvYZ=S!-R`g&0x}O{i5xOsPM3ok7YaA_GrL zgxh%i#oOv_n;+2!@xMZ6`HD3P-9O%wzPeVaLU)ENdE$@1+*P$uj*6!?Rn262N(r$A zIa$e9Z;@)H6Ob$WgbJfl1^E)_SdP;OW9?Kffjd*;^cuTm)2+omxnD?H8FLTl+p{P2 zK`IOmr8|$sO}1Ybtba0VDQafg?Aal@e7WvrB*1(aNdVZ$7DeKJJyv-9pfyNcHinJ= zuqv00Z@!Bkl=LQ1xyr@3PrvkxgQ{V7B}#~2LH)=~K=?u4nZa$j#Y@}3Ve{uVQYv1` zm?v=~A!ZSK&#pKAf9T5Y5xT0+X+6g#!qR(|t6s_rasvfZk>FUem%UU>V@aqX8>+o~ z{ZVr{P?7rPz+OEd1v-VqS_)?29)Y_zCNC-qbrHF|bc z0|@IaAKPLg%bz!1dOY?O!zr-zK%MHJ&LpbsUXz+x-?QtiBSez|IzL{d;lubZe-~&* zdyI!+MSE6#--VGR>~r)b7B3W)&Vd$4if$WwA~seLf*qw?*pQbHN<&874#69ViN)Uc z3+U6bz?Wg z*M})Upy#-Uz4zDb9|;U+(SO7T_wi0X0eNCK3RJ$JFqn_t3JLoTo+?I5DGq{)fL_@W zl!!y_$Y_yGbZ{z<&801aoA6!{+dsCS&cCHGn0>ZyrE6mRFO{l$)i!qQXOE$N^74*? zR~I)lq8bGx*H;@>hIfq|JkLmA^b)~GATI62ddMp2$nbOAB>f+-mS+ud8LF9wZG@vm z2_&;GmK6fK9#6XVdsTJ@6Sy{ju`QOAo!P$TOG1oXCoz)%l zgvbpy5$Z&BHfCu*d!9Edwf6A?LYJ}0g=1AyZgA*z3;6$-A@#zd>sHX zjrU)|eKFTRrbkCd3;p1}u`s2|<}fpD@8q2nUzj2sQpjBw8fO|87m++1hPZ6yKA}X* z#^6M!UotewTK~VUXp#g;ogsc;u|fB-_j|)!OJ1F_wpH+wka)X;)#O0^tj#Ozv9Eu; z6!tP)&-Q&j<Q6?2#pY)(dZjF(su_5fD(epC922}!==L%tCG`Ep222U}*;4%laY zi|#WB9MwcB;bG}^T4VX4)ua#TU6}crOP!x>J)HNvF;0laf?}%@uM*ArSQyMiU%XG% zrLLERF(lL(WJ8ioDSC9@J>QSxfgt1pst$#vTuK6OQ^c@B76-A7(vMW0-=%QyxRW4p zN#g|I1rq3b0R0D)?P?D%UR3f#8jlYG_>S(}|g!=>fr-%uMPgAx!Q(V+`;f*9H1 z{6z%?ef0dd1>8v14vGe@26H~`0%Roly2)bHlMio=hs~4&Y?FSB&skY_{y33@y|n81 z;j~E2dNvWf?L&bNfvGb))>0S7PW=zEr`JXs%_tuuw-}Mii9Q>PyOqfEST@4#x?^^vy}3SK4a>XDg@9e^f``` zCYARpBL$e;8Tp*{C`VVjq9}IbxOmx=l`%(y-vQUnNm)yTP6WeeEmzv!5u0*MPQO~r z`4p^Y18xaLtxc6*7lq*_vD#~!XEMvpyMW(q?=^{j>e6TQvQC(}Hps8odtvApOOjDK zp#Tx0=m`H*TK|~E!YUyjKjxEIqt&C6qcsBhCMPvZxA4`=i;Aii^|QIMpZ<-&_|~^g zv8Q$I-5*0!Z&>x}eSJXX#sOBv+C!Jl&2|kbVwk+aZr)Tq*$=gMpFs@L}@%M*Xzej)5Jez^O$Y@0(4U2_K)6n-`09xIq3c{4cSMz zN1boo%Xou+sig4rmed`I8~Fbp!%Q9mf)20^rW9@u1NjR7$_kcP{jlL2R<7sS-W0Wd z{-vBJ*~R;R0NHsSKov^ec_rE0>hg`9m~SIk?!=5tT<4Va@ok!dQStVxV@S2;VsSd` zgd7bKp)Giu6a_V?F<jCmOUl80X3WG>GWi z&+y5MJy=U2i&md)d5cc!}ahFH|UCM`%jRzRfcyHu31X|l;Kt^>0MDF z^*-0mR8&ZQ7(Kt#FRX@_d+0&)iI;b0V++3m-Rue;c8(B%2cK_v$%P@b5L!gRgx_c7 zIMDy|0zeH(xR)UUbO{ewRZ^Kz4kKVb@cE$cjnS>QWv|Q3e0|Poo%0a-K-zf?z5uY~I3{ee89tWxcQCwkA zXXnL67`dUlzYqr=YMPhJ)!2KZcSTT6tB?)Su{d|Fu{S|q1f%&y3_4m6^>549h__<9=k9;1NGojB#(6n5-izQDZOLl>D$G=t7w+wJ_u05~ zj>4*OyANw7x2#1si4Gr_#=%Aku@+(fR-fbZf8M)@Z~oNMrc=^@@M4v3M1MK|Db>P- zQ*)rHTdTFQoV!gt+NvvXBD(y&=I<&anR3C@!ulnE_b2BHu!XABIK#DjMF&D15En*8 zzB7a=i*L~A;|B(VlL>kR06G}t3kf0}#-|Uz>bdu>rqvwJoLVN)S)CpC_y|CKKv6&H zqpguN?*;RBoVYREY!TSgg=kSVH8uFb7-3$U%uz2nv3Ws7{$-7~v4?{w^Q`@mmk z?bZKIv=NrMlSeU^03eHtzw<_sV(e3l*7+AyoRjctzzBNCw*+UhA}04QqDJ8f*(X5% z>=B=~8>UGT$Ojs?7VUwB#e(^4Pyv%1STUJTI-E0hYq2J3x=1d@Pl)uVqqDV`M?Ft*#DZk z^V)xzTiCwxv1a~zSPM~!{T2OvC%lAnLiFS!Y6h|>7ceVPX1c=vhrs?1r>AF!{c{n= z!_=9TJ4}q6JFr$z{(5H@n;KpiJ>~+GFL@++jx*3~3jQ4E&i^51E`3wukl_5avLN}a zzTCdGml8?Ml0f!hxCu3u@05W)Tzw;)SLTo#IQ?Qtb)9kg{;f&v8!udDDKncMno^?q zRJ8d38iU{kHuCX*f7=-hzQgrs+9{nhW@=SZ(`*DibOK=A7A}kEg}81KDXAm40v1cV z@*@~CPDvcYDDjrEM$_fxfW7s>r{+PU==h>{!VdKoi5l++MbeUZWAXheU@aG7l0heihp2Ls5t3=rel42QY~4Uk>{2pi73 zBS8tlI`fReD(@6}RZi7v=%0ywi*sYSaDTw`DGiG9ahb~E#oun~{K5MfLL7rnKFs7` zZ_w4)*ogHx)TOM=5tZWpFM<3wBcg@;p5&g!;py?ws$Kk5_d4TcX?!IT6eaN3Dy@p6 z4`Z@fuyOT1Pfrul$i)|ER>}oargGpbS5qgaCSnqe3egYqQJ24uUdU@}lxfdP@Yc7h zfUe=e3mYjChNTzi$n5NUKz8-%7MpQZ zuGl%@%gX1iM`-V490 zV#92cqiPQ0l0Uuym}fYLO{r%q#qfq~!Gth^C%y(!Aif0j_rQ4EO*zNY-2%D+wuv<> z=U*3nz4zM29(KKW=Gq;vlRjQ#D%g>!wL8`P@mnk-q#ic>w6gvI4ZgZDn7~OHm~IF) z1yFk7LCpM_#p3Kg=llzLRq;UG4Xvc2)(13uVeS$?570`!n*SdX)m_|ynl`fXy0}{M z$fEXGbm|uUit+pEPhl39?QGePt~`E=;hnlzl+Bv4NFEL$RUG3w9v9(M}ihI^{ zSRiMCV?-8h)aAm;gLXyB>D%k+LdoaG*FSX#tjMN8A5b%Qo0ILwUAWlOp^h7raVpf(s@W1Q(Mke!$mzO#%+bc-8csKHFonR3 zmuj#-4af=ua?q;E4;X-rZj(38=OAx4w~!Fj}=uBNTQ}>n7VV%WA>tSg+C^J zvGT(A<74+s0TLmi?YcsPQh{F$sd6Nm0-N4|M{Z%iX&I!{`WjN|)g zan6;73alhu`X&8ZGpC9IhChvk6DpvcQ|pl6kmX>;f`vg|#y;jA79hm*=cKCG zvb4L&h*tOZKH22e@y+{P@MwGpv07xx5w}4=zzYDI>jx2alK#<42zqtiy5OQc_#8D= znwswaaP2%Qy6tMD9M<4>E`c0C?JneOLl0e27wniyP?KbL^j=sX_{o*!x%Q-SCnvJ$ zBID~J{t3YHZk_I5oR+Gn+^^XG5UWPe#>R%Cp<%|| zA2t775}XxOMXr4ZhW=Md8U#GB9k1PhTrk_)@(ZNW=9^Ni2iKgQ-Y1R-!+`R^|IH)V z(^Cfe806>cnV0KoeyI!eMksgLyJ6`&5!5!gk(;j$QHsV;3<7r(XJniyrocF38axKs zaQ-1i^3Q8LjH4g!*NUJb+-5n-gZg&aWaI{9%bX*eDwGY`4Uy}=c_7SKrdr?u^mtr; z%b&qzDXC-zdA-Y^8DK_^nc6#K8i!X(g6sTCj4l;>S@~(JblG1wOgTAR1|1B} zZKCbn0U>~Ta?Qgnl_@GxT!BA^Uu(Lttlw;O{X5YE;L>S-824gDBSWI`q#(Kc$m<}| zd7_%?<7M!KN7~2XfWQC2L=zy)3hwIAT&c)_Ui}7#Z;x)2^n2X?*GgVH-$U_y4v*mI z3iImd!x0gn>#=`#YLUYtA0Xt&D5D-1fL|aMUkDGh1ks?1>{_Zqwg6@VuWkmB4dWo?qA9~oX3e9Q5@kX$Lsi*ws`(# zYw~%0fFF!t+fwz%-*EcZ2yGW{v?@V{f>)I68u=Q}K9VxUmsOb3d!ou^3!WtM?0h0> z-)#K*@47i%#*_rB*{gl3Dzou_{FJ~r>*_4Df%{M>k5(k<34cRfYTDE}3WVA%$Ds+? zED`vP;o%}rf$@kDDD*+`zdf#?4Q~8t7?>H#gZLJmAwUgnoY)ydh93t$KfbU0;aBTcCi+x4D z)PLRx?L@yuwP7Uuv60=6Gz#6nzy0p}R_^B$3i?m`lSyu`*CrI`4z@|#nmBlxNIWm+ zEkhJT-1cVQSXdOi)UB4~LLkvsuegj#i;JHcoU6UM_*$k{_cNlP?f$?q>1TZ>=8nwQ(8TnpQlhI{+wC;h7QxDv_i}s4O|9q#A;#hu8jh?O^pkLh*f`#*HsZYsXbw&#YOxtj$L*%D{6qq?Hzf?B1bq zlhkYTn~wQVA}62Yvc!u$NMaxG%JIu+q+4!+F%-XEelLOy8FdF*UZQ0?R7&3KU!9FsCQ6Ih6u<=|#&Ld(MI9s%oSB zf74QH6}k=tC#SW|M7zPhws<;JvK53{o~pSVwKz1YM)czGL~!@)W|HsS;i$=rm{+0< zGnZMVs>Zd#sOg6Ff!77UsDXzc2tVLKSoFFghS|TeOrjG-W2Ep-`eXVK;>QVpzqwwI z0%q~67%zDFZzpa%Uy0QEn@q(S)Z`K%<@6sbh66^qc`skJ%^Cdocl7VRopx1j?mJB5 zG~>v}a&F%2MzCVFHPEPJLA?r?7@@69T=X$J^WFFK1 zUKZxbMq#%+OeVE4n`0E_=g$MHwzK?~n{H6sYFSX$lr?hen7(v7>)y*}v>09fsC2O2 zCl3e?s0^j^^ISCinv}G#%>_*wBZ(M2Nq-<`B7Vv@5uM{?5R5y!jqtlDH6GMoxtZOW z5zBHp*=3@;4l6Ti{e5^tPq~AdOcxs2*+%{Hb=LEZ{~bsGgAR1+@KfYxPv?(+viEOy zu0!~2O*d}uvR!vFq7;FTv*HbHp7UYUX@oY=$HE}l4`W5I9f|JlR6!Y5%NG8GlIi;o zAmz^?i4NvH?}D$7(TWey!6HH07QkQ5QFwY>viOu=ToV zKXD~mxK=Iqt9Sa?h1?SavdRD-D!mskRnZ%Ghyi5B21`v-5%ckAcy=)z8;p@(p|@qO zZkog=+2g4LI3vwGVM?;t=FRjjsewCH(Icl(R1N!)`}9P*;Af#_us4YOiN&ArJJ$nl zW87&uXq${>$S_l5M%_8p7TBoy>c^9D6oS# z8-{Xl(zD>I5HWoQe~DOMzNqjR{C79@k34(zM}7S)BlyF^MWz3%!(%j}9xVVn{J8+& z>G4K>k`bSRah#XWw1&bUw$xwgNGsD~H}jnlgP6jj4Ig=M-)T$ziUKWYgxIuMZ2fqYFM+LK-?5@by% zbyv0N?7zuj%-e9RzkEuXM);Ctg_``oCCPqA0qr7@*Gv6|x9$IK+fO&S{pT4CKZG{Z z!j{qBUQ2Yb-{sQX4?uPr+V2mf(Yjc(e5z5+cYk!W=l#W;eO6R);pJln!-5YoVF$Zx zV-Y|PC81Gd#y&qf7mzVARE{kZP#zD3;}qpY(JXv}p61)b=H>$|e01_C79At6{9}rk zOdSKKkj{N>pHe}Ni)xx59M$${>MWz{lF0;_|J9Aq`3E?oKa z-R=Cx#VE1UcCKhC_ed{#4&9|m>s`qLZf75~b`AaW-n#?ivgz*xdUV>NhYF5-kiTKd3U9r3HMqoD(CL#{Lq zqn*WyT8Gy?>@E8RzEv$Q@&5~3sC*BJXQr$Qk#VUVNfV~R?6b#)KIwR_+qPJNPj6ue z=eh1#oE$k=+_0eaFt8I6yT^FG^Uv1!HuZv#4L_TJABMYz<+GJyi5KXsB;*9L!}$Be zp_@f)jja7|NV@Cfn+%9&(Gm=ft1I$fNz%aADRTfzD)z zW=hv~67Be_-HFRtwqN+z>$LUAcZ~W<#3yJasU0OQOq}!HEiMF+nFRLremnBn7T+4( zlzH0j0#3uscHTz62jc%7wj4WXSaBfBgXe(5s1?^eQgO{hT(6_VZue`%n{$(?qsOpK z8JdI+*^mg>{wW1^ZNW*33?6>_(Qy0aZ(DqKj0jP8N$f}OYjQjb-y9)?{%Q!BQI!?m z7BKqY+CXfttj2BuD%p=uZ8`jyVO6#(vSvD)9(EoZSmS`QbMwZn?$E(HkMb{;I)pE| za+swIRgLs~jV(>_jUSqMrBOYQEZl9?F1k040{7yxYtL ziJ;<>0Glii7b5MbsN0br9#jnhc701pKc>|3xNkbW(oM5l)LjA#-@ZBYNip&a9%h#` z+9Js+nra`63d|KC_H5h8Q++q`g;O%;E$-Msg+8Z#iBX*J&wuQDSCY>izuqgkziW4Z zq@=IGY=iK_Wn4C>k~W6FSzjv*pp-qNCf}Fs-;8trd=f5p80^krqfo#@)YM{9^ASY+ zo>%X8qk;iU zSkWzVG!~a3k@8Y10&HF$63?6OguVj+eg{OSgY+^xkC+uA#7P?6d*BxH5V;T3@2M9rpKA^!cMonwv#7k3?yL!g&6sKN^BhH9VXYtaF3p)WOX zCU`{11-v={hJINkB_(!3C}G|mt%p6r_1bV-28_?jyWl2!`n+ECidy@~d%2*51gkp8pSXzwkQ#sNru*YkB-4%xT4-l$F~56a<7?0T zmL#)7MUen-NcAAdJUkzOM~thH4JA)jI%$=Uj?G5uyID1mFxBGDyEKFbzf* zf64q>NHMGztm+bU#$WK_L*}Axb}Soi`uqlZ0$G%8^4)6vrFOAn$@qPPQs{dX`#H-I zJy$LU^x!?=lp1kPIg@JApH%qx$21}#9CZ+*aiOoTo?lhorQY>_*TJs2n|Q?C0;Dwn zu=tFOrFpVAler4R(+&(m^0-Zxhg92&KL)h-!FHo4%My=K8|y!QkxUS*2A*tED+>I? zfR$QzV%s6d(rg6}f{gqUE5;oMeA?t3K+7HLkL0%dKBtWUo4GC08`< zzG%L9m5dsy?;rY@WN48s1~=%e_KbzlQ_rJp|BHN1B8>%-c=vv&rFV)%JHap}QTzmD zC2rO>D2fe;NS0n-(`@P6Cd}ep4CrY0Uwaz%{(Jlns!E^-zbv;Il>x#4nnPEBB*2-) zAR?3(Kj|SJDOXyQLyB$5fa6lFk~8&=y(SU}I(^pMVC2No((42Pp7toEihnYXI42^i%7Ep%N19l7Uz91-dX~<_X zhark{rEl41lCe2u=kyA-qDiuI;f9~lElwlQx;OaA%&iJwwh3Ec1*i)m`oh-Df22v@ z?MXu@=TPzvsGh+;$rK*nXGOuit{vJyTCG%_405A9GbX$qjJFTnVs#=D32~hIi&LEv zRPQ_&L^^YFDW`+^|85(NC$V6vG6!wOASGn>A-nD$@NcO~s!;Eht=0Z_$`e(g>jtiSn6l8aKIc=cTXf1;znsl;Dcl1GTJ5b@+Q zRgl}|G>l^^hkI16GSW7A48-;8DcRGgMppbl<u;uSgcP9n|olWLXy&qp@d~Qd+ zW0tM@-}?TBb6a&s(OTIE=c${z=t>DOFpOM6Y`L(LG9P2LQ*-i`;u#Q7O7qjOZaP!c z3!jw36}Y>0mya~a6$^DD6c*4SKgScCV<)fBQZ(Rx|Gj+jm)?tS(fZE~U1CuUZSVFZ z=sD+erSDetueb+a^%!(-;_K(h@J9x8nUgedL}AIy4LA%vSDkBFXO1o&uW^d2lr}cy z_yWeNd8yJbr}4@;F+maK-Du|4H5Mb){YY#xY@+ysLC$?ydAr}Wf8a4J%EuNEDtAP}K9Y4Iq7eckMF6hp<|c_c}eb9#1maERrnGk~{MfkEAe zCJQ9h8i*$Kss9y;_CPO$PCvXv2L8PfAF0|&LwM3e$W?J7N>=%l3Ihi#T?;4X?-M<> zyO?7+o3_RAXqS#w0-}}! zP35FN%<_;OzUSihfql|>f6Sg*m`BD~Ip9+m zSnXNy(l3^AnQl&g`J!kdM%g%)Di+1@p+RMWowZf=I4#92`QsN;+n8Xw4X$HLTgZlrwJrO7nhJ&GYvjFGxwJ*%_`?PI8y;y)hjo7Pb9xnIj1d{c* z2Gv}lk|(SXBj)AObEQO_IAAC}aPDUE^AJ86Ar6KK3Vfsv`Vd9azmB5A@IPaeu=&l* z$ye|Lw*oy0koOu2G|wM_`^M5Nzned8?wgNTR7D3$sA7i)j_STo&CI0bXW>c(L|$h+ zIrTs-PO#mBIc?fIdg~A0z_oHdSlw|wTY@w{-0W>~I(kLEBJ=~q1g$Ww*)gFVGAiD5uQ!0f7c@Gjm`-#J87QoGokFx zO^JnH^D0Z$JhGo11L{x(uUyh({UvU42@mIbYS9L$IW;`ibo}u>T4~0V#8BhJmNX1u zQ_-S6vXZ|`)ITT`EW}SS z_9~f@-aJB!DCjjA9gf?<&68kf3S}Ez;UANjf;jOQILkQU2@43|09Te#V7(H+{{t&q zIx2>)z!0m5195;lYlf>P?1S@3FBuhHA7_|Z3LcXFMWBy~o^i~R&=cXd6MT$aDHI;D z5yd}Rb%%s1#7dkhX5E77QEFhQ2nn(VzJnFf@wB|F{V4mXugr-olv8&@+O_9-l7T7N zXFxJoEpl{-MInuil#wCK7{{*?HmQJNawwAxF&qD7QANpu!3EV|JB{F|sKPat?yS$e zJZZaX9gCq8=5ODgfRJWW`&D#rdcQxx}ZH z=71QVzs1v#*p3Dd;3Yee(@ueGww>M=F~geSd~pU_i#!}drTos7CRp*bz+s|Iv@bm< zM`L?V@MvbuwK-4V`BV<=WWw2VL0x>^kT_^&?BrFAI20}KF&usn%UAbTKMcrF6&)T|j+VVg@2=KMXo)X0s0-w$nB$f^w%Xj>4AhU^i8j99)yYsFBHfLJMDCcG>b9(dpl2v_W|`9_ zXq^zG<#xdFD+zUMuZsa>GM^&SO{fu;;Q{nNS8JzK`K#W@3z0>@TEDa^+2O&+LLJBd zE@5pwpz8C=bd(^*T>ktukV-nZBa$r6CP%%;$JHSq%B@#hRRJpPs5g9_W<+SYU5OGbuW_j2tD!XR<_JD|S9k?S*!@A<@5OxwjQ%2}dpSdv(m6m{qT zY8M|=r5%)(dc7aJ9^tTP9pLZ?ZsK=gB$06$Sn&$&bZ+;BUK~bs4!?_CZkO^vW!ne# z@x@689#4I1v*FJQ);>Mgg~Q_?OYr$qKrwy1cQaKFf-Z7#^StG&FLAB+0I;UXR64!%ZhK8-}cryEbO&F~j}P;^3_exsJ=a?}Zf7J0IAr284L zLb}7&mFk7C0UP)Ws-?k+F*N02dvSpRk(OvF)KCCD{~o7aZIMedf)r!hcuoe^S$;;1 zqS*1yO`0q4S)Y%G&DK>`?4hhbp=@BBhgX-imB=n2nL3~`>w4I7|BN9_f!;Vdl1|6a z?c3&-{8jhSuRtK~OF}Xp>RSKX2LFU3|9`i9wzDtOF7mm3-`LpNZlb#H{6)|8XrcJd zAs`iiuj&F-NzkN$7+jUdBMMG&Lo_?qKk4e4f z32AAzy8DO{I`4hn=G#DVLO2p&o$Wk#IpXLyWy<;p#e*Dnv!7C(mpl4$^a;|uJo|5G znidGds+`xRsyPErId7u6B&&%F=Eh4@e=X=y@#WQf72Z8S(3D$a!;kOqZX{}ekYIxi z#r4a+T5$)n0hm>%omdnAKbl%KYX_)!iGV3uOugG(;H;_Q{NE^q#aUl~F{k2V<=V!; zJs+o7rS4fc+Hk7McQ#=S6YGi%3+1m88N>KxavN5h2Tcy%xznLP!IDYVTd>x5LU!+A zi)F^k3xY8WbJd|tAzRCs&+1Qj?nSIe{jHWu_{5DXK~EL}{dHak ziukx`Eo9d!7(n?T*^}`jkexuLt&7<(UFezze1*iXvT5r(hGOH641=H~(_fm+wRbSo zKBJ%DrN2x2w%$kHD;&xvdTL_F-g;VE#E!&6QWpPwSdP8Ak0g-|g;;+d4jlb$$;SWM zncoi&E9v~_AKFTu&r0HL`sDHcyih6J+4L9DfT}^MIqp5sa4+Sw>rhtb#0Dv zNM|HIRtILb zjtJLH0u^)%F#h0%$g9@$bkvUG$yE5Ls_D_ONR6oC_1yq`gblpO5Ts|u3HS;C75-J&X#d0 z%J=f}!^J*^PmBIcBqXCpA-69`YHPYuBNjJ!PUbnn)HgOX_kkk7^=!iSM4 ztarNOZIy&4^9yt$)jnjDUEb@cZW!cfHX);cc&mBAlpK~SxX**rfy=T*s&s)CjYbrjwXiv#^*5~O6*g;Z@+OFS_)3WMG z&hSZmzn$%pHzRbH_;=~vBrO+bB{m>-u=Q*t?+_vq&?$Y|U+Ud_3{pY|Ws_2%T`CrI5 zZp!AL#T6O2=hj{)>Qj9>7ktQBvbxs@OY+u=>MH>+&F)EfF|0um&s}~HS3RQr*lZgV zSrC__Llb(A>Ol(#ryKal^XUdixJpG-_!*R(<;|iVbic{(7VQdjv!JOX+amiT%KVfLlR&-f2^+IyZKrZpb^ zblm&;Ce#^|#=%U%rsYM3=h|j+w$EX5s)^}JRk8}6F7y|klpit#|- zy{-ekU;N_+DwnDC+=-Lzl_QMaA1%1`?j|oBngJI2uHip}XlrP?V|B0W%=OKsw0$>x z;JJoQ@$OM?(29f&T;P8tjxgjjqX^wEsCNtMBqRCI+1hC^wIc9SgVmdJEYf4?c>(M= zZC$&vN&OJLFeu*jiE&55Cy1wGXKW-uuPE-NNC9<2CQA6WEfNsSD;JF zE7Nma%03O7pXA0QR2L-k8^`0|m3p(jLb$(UtXF$&=wW_ZOJkfMjQQUu`I~0X97%6U zk#Tyl^^s?y(A~G~>J8m5^4mPiW>IoTz?cu6yHrFEfHo_P+c!~+NL+@RKjZPajfmNP zJ({r5D4s8oT%+#UEN0dc9~|GVWbmWqO zgm6JJnYT@PInE!m2Yzycmi+WF{&8Zs7ek{Dp9|TDBJO8|X zj1ugjMtNOvaRvhpA(7j~Z@U(Ym@r=9r(EJk0-u@l7a|%H27&CId z7}FGkPD^KrUC}F?Ynj1@htGJGmg9sqWTHG?qivab_V|SPU9Wu}U%)dhwlXE%POPgG zlgVuIrj`k^u70%bIgP~od4r?3RbCOP-`Aj__O%{8^!xSvBJY zn51|`j3V#9N-bmWD#f=fjgJhKOC%thLX9{{uztez%C#b%sr(ztl@n_GTKW_u<42G%) z*vqDN8jy93{s8H0#~zP2e|^gHl~{7rb1_v0jY})JyDjHH=Wug zXmRA2RN=F>uZtox#1C zAyRu}YIrle}cf6GU(JBXbw{qmN&~R-N?#X@b7wkXO#h52M=D{LC(& zmaeF}o<@l?*zE&BLs_FL4eevPM!twENA$PaE|%4MD=%G;*O)TWniU^yvc$GTMcgls zWw;9a5G`flHBXhPj;dF1*?I0vA!enmx&I$YSKd=E+bwTD4i3U(E^dXXmNu!Wau#xu zd&f$@q1m5CJnA9nmZu^$r1@oB)bA<@>v&>d=I3=+nrh>5ARPyl#@m-my`^B^zx>c% z-nyq?ZfmO^{JWu;r%G1q$DtfbtqDBPr!Yv42uqvm{_AGdC&c%Mi1ww;$65GuKkACv zdBsaUThEn?&)Iv$!RHtEa#-8Ev9svKc6>4$PgxHxQKjX1H~W}eUrHWOF$vwqCtu#a zz975}^MIsX8Z#e@tWN;Z>oN|d-akJx4apr8eSLTFQQTp52O1&5O54HwI6 zraF=GUri-5ryZJtHDW^fDw}$b3cfabD_)-r_hx{!_?IQG9VusA!tdYBlU&pL?v>#r zz0g%@Bzp77u1qn>KJPX3J}6Mz==AiFqrBiwiFsvE=Rr()nTbNQw-M0x%8BsEh97x@CQ_%z z$=xX`&Vem=-$`F@UjD(u{f^B3XLSSaX+S$4xbMlcR^u=hn8)=41OTEd@Ly-b^i2iKnH{mvL0qLd4~?iwc062kdlj zI5vx~R9(r9R+}8$5*M(!adi1`g=A zuln{8yfd$a-Ukf`t&E3{2-Uf){gb8q?V$)f1r?nul~_Ud-uJ2YGU`kdYEMA>EDqw_ zm@l;|QegbrKRq23{X``oX-Z>&*H(Mh<`YPeX%7o{Jg>hL?uOw8&=bKPvXIK(_7(hg zmDe$x^4{_KO_HpWHiKz1fvGMN2zd^YJ#M@H02zOE?Z9u{p_k06Khd~#ZRDyiL}M(5 z4}|af!_Iv#M=&a=QuB*hn1(SwVjBLh9k(bPeCmBO=Z^(EOvlTchR#X;hGw|@qywM))(qQ~=`TrOm7*dC(IgZ-xqG-!EZzeuS*I7s@$$T(Rk&4TSIa}T?G z6x;8Yi7gF(N6QD9^`-&G{re{bkVeX7N@s;4q-YyY>e6?}3yE>fOU zpvU?q&OqEYr4!nfHUW&-?;{d$57x&s;ISKSJw}Ll6;LC+YWHoesjVgdEaWj=*71$u z5TqRYvQ?Hys;fcx7u3a_y~zIa<_{s~-#2DWDPUQ!>IH=-})Jq-AAQxX-G`B%8$YJ}?l0Z@*z}9WYsOQ&Us6ULK&0zcJ8> z9!Ds6=ge2#etmj#GG3YqbeyMV)xzc4uU=W)P=baIK69HtGw)hTM=H~uhWV}+H=_mi zzw6wMXWpLpPK$gaVrg8*pC17_K;JCgK7&>>6hVK(xkC}`R;|qknfTS`1HCrLA*ous zuQ#v%&Gz)ZL-2E#X(tt`Zk?yP7xci=wF0K^wnsu%0c~7@iZc)mg=3rr8Q0ZRIkK)aFI7CWC^ZAT!M836TUMrUJVW7IA#E`guiA*DbVc4c8_ zM=8xKWII1PYIt+6?RpH1iL&;I2HebbDi98yH_ZYy>}=z$Xzh0!sgt4P)y~VMx;J_} zng_3(Ge2%zHcENxn-Y?da4sy?e%jD`8*>P&%K!WODeG`og_g7K#V-R2vRRm4$6<+@ z&(n9sJ>hfAELaY96}YfCRUOy66yH30NzD4@YOOZHX`*9+&P#I1eGTgp>?m#hqHAg= z6xK*LPKtt&wdF9-d!2@IRzJze(!=-W2Y~;=f3LDZ1)(%WNpwr*?UM?tX%=|rM|5=kZ+VwyK6P^o z3F-QOX}=t@W8zs2CXk;wz>eT)*v?P=>e2!#arxF`p9TM5X}fqmZ>B&V>)^HQbKVam zhwF#kIQ5vz@8tOz_jg41fQEF2Y}o(D2cXsO3c=Vo9%N1G>Izt%3*@}gzq?8j1gGyl zsgkPdgbkVBMK>kWbQ0M*3Z-|&yrb!f#+$i0Ef1iME^Qa`I!m^tyq@{8dPe?LVhxti zr~?G#8C!$*7;(;cNep>>|mBeoW zMaPvd5Pbm3rS3NEH8316X}S2tosB>95PRPf~gL>?-*}?EJRZMyoe2nd@J8I@YBLo>%?Xr2c8pe>>lR z%B+s5G33UxvdOWrhuYm+Ae@fFNf-8FBFp*R_9e<5Ns!|;@p2Yy3ddPAh8r8H?F>u> zA6QKktF5GDh^?pBe%^Xe%k&z~vgNt!5fEt5d!6ssG8v^cHaM08CC0b2EzZA|z}VmK4alh0B5c$e26d{P4zYi6*fFF#j;MZZRrc^pmQZ2ABmySgC^Txg9ppx%|W zv@B-ssztB(92ezvy5x0$#k1|*0xH)wsxxm&XdgYlwMcVcjX(~6wX*Z6d|xN#rBPbN zQ@4oc20l5gns)@57r~`c&RqsId9q1bm`K-%-=6d8Oc_5}`~G(N>JyUfYxJ4yQ3c;) z0w;~}gOXR(>oAWpLFcItle~69VdWm1rr3#KZlWBFzn_=&=N~q*TO`tadc?(NwLRU; z>~8$y+yqu8mU_VXcl-AHH{!SG!71~E$mQ|m8*kaWMK)u_FIO9{+pV6O()=e+2?s;h za7V~Z7P7K$T<`>4f5iNnz5KbQy@xQ0@zQ_5D4H*Sx0t!x49m%~(;)xS=H z_*;g@CW%+nEgsXG45<^uQM3_}yA~olBN9B+r-t{=Z7Y18oSe+K#GjI3o8nx=-QcPO z1o4@7IQBZx8hi+kbTg%GVJ?QfZtL{g0d$W}u@{^O3>U+^8Yp_azrAO)p(MY7A2Nh~W z!;{OGe71?bw&nqi1shTK%5f9;xHnBir(Kdm}l}S^X}IcJ1GJAXcD!JsLaC;^+#HuMrG)c ze!bPMMQTY2lQy?K=fOgw$`Y6m;fkPfrBh`bAvalQ_upOs-rt{5ay;Nf+Cr{giF%Ym z9sZVqg@p%#*@2vPJ^{`$khHR=bwO#K%XG6{cS5PO?Y+U%4tDhI<^=8?tnVbDL@BeoU-yy9I zosO}N57VC`=lmQsK^xRL9!YIl`sYHZ6ag;A4PO{%=}Ynfg?@%_9Oir9*oQCi*q1*a zd2=pkk0)|_v37KSaQW`mE!A&5PRdmqd*a6&tfcIX5FGMaa9#!JY%56EJA!(I0Z@;$ z_<{6m-Djz9eZR=R-d`uWJBsm3Xg+A0Z(5?n-7mjShJi)zIE{Sb1LV!GO;svlh|yki zteS9C4;2WXGJ-_0Fbye>b@=R*{z*wkGtNyS69XT&X^3S==XtRT@zdW8fBt#t;>tP> zG8Z153lp##OUArU4hLD=k7k!nzpZrW?=FOo-OkhTFqA-f6VQMqjw^`hQutmN7L}LB zyuyR?npD$(Ri&t>wzq$Whtm9LUcM=Fc4pjk!mu>J60q$x5cyjhNp6lL>p|*0v9?)}D5NW&_ zSKBHfP!JqNFTVj={8&($rVfiUoGMpOS~n84Y3b-B_nQxd(IN+3Yzq}&Ir{_^d3MdT z^mH4@yB~0Ql$U){)@pEP{q&hk3L-Z(ihEcUm?b^BQC>-8t0^fD?+rdRjz}Rc36;Hg zuc23o;70giUVpdzJ}9<7E)cothhBqvVH19Y^oX8@o3r_lRnK*LDc;Bh_}M!EebLLT zoKx5g0|k%!XwH_7+ipNEa-TivX%nm8Z9W1TT7=^t$&@CYvx%}@or>CHIw$l@w{;x; ztxJM`^Wl*QcbaV)w!f^-1tdgn_Mfe<=k?H-BnxAM4-@(m^?VLJ*F z`IXIQ5TOO<883x_nZf6jp9UhcI&9nV2m<{odB4e<;? zaQUpRzURk14JJjx8P6Uf4NdvOFCw=NFzAYaf1;l(`Y8(l&rsj)hg( z==q$g!GB(L?i~_H{sHQ6e(c%$IpB3h-b@CP8B#OK^Ka9_7ZY6in?5aDYb=Ie`u?-F z+((XKBJ9}}3@-?R2{p)z2aQ`4gF4u~YcW0=5E?p#?8DReOm=S5k1lx%eVH1*G^QE< z8ecaMxf|++==dGX*>*Rqf8{$qGkN#F;}wR0<~i^8#eQ$*e6||lGd5fxd@;bi@hvSa z85;ZD_^S-b;hK=2b*wfAORaT7@pQ6Zl@`W%Kwxv9G{B13{w=91ZT+ofz_CdfFk?^vH)Ve?>2>5H+nUJ8>>Kn@r%Rcs6SK9|7v$%|WPkfSR_| z`ltGm*u=USgxv!HlTaP+tGyM9jDuwuY73ADK+H{MMFIRj(4|CF^|K9nhyedJ7pGnzLIZG>o~pD@on^T!*(V)h@l{1|4pAaBYczy z_z*J4q|&f!pXs}1>ft+R^Z!Aap%MXADHS?i)5W%I7o}nEpJMK>35beaYumM|xtA0c z#xCEU)%z@rX-q8G&(GRJdu7J`>aTt*C1G;@+#Gshwm$oTb1Qwm)DUFw2UHNxb z*dhvEKSgT#lQ6|Zad8n5;s5#rnIIr^(^V3AD~UHH<%0-&8by<7B%^|>=X2=C7~!0O z0hy+Qwrf<^x8Ymkvy9jULNumch(~D#cNXc1%-YWn+V6yQU1v?E!i<+g@NmUvLRi}{ zb#aL!P``$^ruhU>R&8Gupk(Hrtd?6J(>?n?e;zTQQygvpN^@D8zoW7mv^`3ipK@$m z;D&g6y?j@Uz`N*7pljcWl<@w&q`vQ^$pL-;s4x{9D36y!j-g}~p7KHXDv92BcbwGj zy&&29xSJ3xk6sMIW88#3g`#h0B=bb;nb9*dr9%BQcI`T(zb*auMiAR0`d+&}v=Py< zE#{?~%cRifdHc)f>VCq4bJd%DG6B~p3Q&gkkkb@X2+%_lQ&Z0={eH0nOg9Q5rD>YS zwY9Y;ms^=IHi-TT;02j59Lh(r8Jc&XRxUd$3ljNe*OynK75LVyBt6dvFyLo-bU0mr z5d!h|_VL*U9XGP7syI@FeVQzBgj+@ge?}d1vOIoFeL?DX+)uy0x%vDZy+Yl|b`Xwm z7Od#=1dt75#cH1j;oO!vHy-WAL}o=4JuO=g-I9u(lQ2hsBfnju?H3 zCK@p#cpcqNBep?y_wWS{*z7D6L<1tjAC!AP@I-kt&!~G42OnHg7`(Io>k0h*8lFlb z-X4$QUA@-GE=mkHu=Bi{A|WJQF&juqU&4bUiQEW4p-{}F3D?7o8X6i(h>H^g6^}eb zSzkeEquBQL_WF+>rG3V+sfLDz&H&+zfqR)3J}0o>(CxTiTE8S?*LImtEkl5h&mba_ zIX6GQJuGmbinw(;tzYgP<$&Pr4vTbM_5i1%_?hWAiRqO%kVG?5xs&dMP9jMwKxX0)Ee~<= zyBg1Yyu2A3iz^wuwXspuZGH|^Jk239x2tbhzXNWj!0VtjiN|J6qS^HawTOs_rM0!{ z&C&UJf!mUYlwtX9QmqjUBoQy#$0ye<&>!**1L1+ZXRK+D@3d-8V`=o#;yIfd7a8Zq zU^E1Rii!X}@|Pd?K*>y7Ulmo5PwV1#<*rIF=01G~gx3YG@c_ihJ9`}+qD{vGPl#S& z;7{_eq|Mtl%f)pW z3Qtr)LiF8NkUTF|!X~L=OCdp^^}%Zdy!6Dx{Hr4nWW7blz}W63waX0+Mb^c#nXi+J zD`ID7uK|PGT;#!o@M417Hq>0R^nKt`Cp9u=6%-kR4w0xzUKlXE;|^%HTNch!eon5) z8I+7nvY--(Ph4!x%E}t@pwD{7S@Z`+2q(fXF+A(^tk3)3zbgjwUIm|(w_RCE+Oq1m zEi^Uk39uoEqY51M#Fe2T1a?H=HCWF%rjWJHs8%#p#x5;hvrX2krW~6{{fjMA5rdYx zG@jh%ukok}x;{gfi^2eAH);t_v)Q$jns4jthi8n8jEP-}KdD>67RsDj@^z3xR1KHm z)R<5;ZARcGf6Uvp%PD?(NRZGABxxVus)#pY?J0&u0BIK0V^f#y-MX1#Md*PkixS)Z z{lOkX7GiXCwE9+uojYk8mbt0FFeqBj-Qd<-P5)Npn>w58ZnA>g9un6$#xGBZ7l zvN+#gw0!W`Ft)fCvFoIHnR|pf#Lmvj>WQxNs3c-YnM>e6LjfTrWqF^*ngi~bV!s~n zMu^(9F-g;@6C zl&Q6C&GvE0i|RCo632RFbbXO6g+#&Ge>z)W>ayBVOBOX=vfi?GZ`M%yxlyNoz_)nE zN<#k}_yJ|4m}f6=6JXlFm-JK&WMd+CmkK6S**KQs#BHr-@h+lVx)jA)UPT4o33BrC zb$m!|z0ws~6^}l};dBw9-N*67$|2m*_Cj0gIJ~J%0_EYT0lUg7D({(*cY@V|?ZHvy5jX;m)w+NVyCcS81qdb)LD}T`nJIr@oC4{ER4V_R* z?<^?i@fm9Fn}X!_r{~!+jMV?fCbC3zbS_wWUX`knB9@7mk?H#MYNmSke-}Uig>Zgq zD=^mI8ODJM;oUY>=(z);lv!*z3+@&eYi9V#5(Pou7HvqdQHpfk@buPcLV1jKd2J4H zTi4+uVWqK16LPrWCP9RzSK1=}hmwnmOVr8adn`2??8ExCmhX!-E~I0t^V!v!xKJuNGtYu|SBKvy89Rc@K=rg;~EhUURP-%o%q zRjjURc^|ayOn%%Wi8(y)fVs|2!{9N4fDm;Bu$?9Z@LdD}llk)LmTrxpfX zxfakC<^4Gz8fGVOUZ4IP5#imSF(;eo)b}}kk3M0_-zRXkQ^{hqFg=B%$TBYrrm}HU zF)YDoJs0BJro+V$%Hy*i^EUFA7i%##$DnSI^F4|<+3mn{4q_g>Fl=(=FSSBQt~Hs* z5!SR&lPIM7IENZ_@~AuolV7UMM(Tu_uH^&$al7OlsZ5ArKJ9Hs!#6$|#)b6p+FJHq z{TjM!wO#Ak|Fd4OIl-bY)Ch!bTpz1Vrv|BJ@^Yr^H*(FYs=Hpr_^YF^>NX{KdwYA9 zK#M<>TWyrr zhOwf<;=;(UJv=S`E@M165Kn~!SVXV3PA_ta#?8Z%5-9>|D-kmd5o3ddLTI-nI+(Hn zk{c+5T`?rQWLyuPYh;n0+s<~bx3`~Y{HQ5M;8MuzClU_!1mf8f6ifCd5AI0uBJl&pn@=%O>g145PB<(rUhG~xVZLYoN_nm$kbK2-xwb*3H%Qyh> zr>yA3906rO1yFz~eefK-o13TZJq{9(d3fIY|u6>QP_O}i``TV5c!rlPKX-v?lU$k+@{ zGiJz9JGM3@}-pSP?^g7W)-(gnsN*pP>}) z6YyFU7*j^|jWS;n^$mwhiK4FH`JtJ>!44Mz!uS^*nIRjO8ZdD|PVzSC%Ib^|JFDnp zQu1-S2a54wK$@i)MNX95-#~x?E^Y_sI>ICIf|z=&GH%o%K8g2`pqS0+l&g4r%9k;V zY?BG}0`y^UO!&COwtF*`h}5SN+h_B38tV*w<2TpmPsE9~Y?fL+WPpZXAU*m3EIZl& z>uAoZd=&H^pp--<4a)EN{Fy=W;CN&3Jw5v#4M>5iwxT8=k4(|_ID(Jt=4v8d8yTTR8fu#QL%Hj46gEI# zjyMXR6D1dl;Sf5+UQ`nfY1l>26uvo82^@G}0f@8{VD9wsx*)DD`b_C;)(|Lp%LhQE z>5qY7M&cbvj2s;CWdKZVOqZaUc3V_!OQ1%*_9Ln$i6+335B+Ud;c#b?PtmVY+%9=Z z@ER@lml@etosAO5ZSQ_9dl6B2FMg;@z^D9Pvz8D5@Yi3EDJB`hphoVe0bjT4mq}E? znZ8fn^5nfw5q%kcR#0G<`l9nTBj0ZEuOa&ReTrxiR|f#2{Eg?{}k zP}o=-UALZm6SNHQB6I3FLJzXS!87Jz64GlmE zse({Y*8&QW#nuca;ZpBiEIw4x)1&BAhmX{m;ekv}Pmf<5tyut)V*nd|;8i+q0#@R053QD*dH-e#_e=x@&x1pV*ow54al^tZ{HLrl?pBK5V7m< zQTtq>9q{6CP2>Wu44L|x^8VjKkh%@%CU8cUVGXy6DHU%`jg1*tUq{g}F}-{xg{gk6 zTn;gK>#zA-lZN^_C|cC@QvD%ioBg->IAlb4FVvRy1G5D6nqFrH4oE~|`WXU}yA-&s z7GKOs92Q?y4s&sly!pE+gizTJ+0}naNnhkx9kO%WKCR{BVYmg6Ck&IS)&NY}9SJAs z0_$<1FiVz2gUW>x|6n^&+cq2Y<(nnMWoBlULo$8B$f#rzwSl*eWy~Qx+bN^SNW(@Z zZBA=+O+`iZn2kK#_C;WaA{B_oo`uxZe8ly~=d3-%pbFkNFr2~FMARD4`1yBDCnkbXi69e^AKq+m0{VKYzq`#^{Ks^%G$DT-9(y9r>Y6}GYuXZJ)eDa zWpMW~UvL)hpNIy2>zRR-XQM911>yz3bklm!(fVHvS_ZHk0b;o<(6;bHYT#eL2kk ze#+$A(SQ&ak8|BA6+ma9WC01uZ;Faa6K{h2tSm`=yM!s=gV$h7=y1W67E=F+oSLFN zUwiAiOo34J#gfrPC#&^Y8l`c+Ld3yiqp#Iu30VJoB4RE?18n#aXjEX?vdiMPnqULZ zy->o|Csw~dqJs=}Pj^!pXIXS!K&yX_Mg06x?RRkVxQVMMToE^;TN6QlLF->ZA~Sak z9-NGrbd|8x+Or^X9bFHHvgmuoHq*cJbTbyo>R$ zFbgNG4``u$l$C!#UV-!>L+Oi%_=@lbgM|QvYQUp zLD))A(?pKGln|D2U!v64@{TuqQRbTzw2H>Tzv!M5F|%x`tc(MpW?@4^GO+ZM920Bz zNXQ;OlmT2EArX<}=;-MB{=V?UV(b|*9(-(jK`|V~=lu`GQ=i$Uw8WU!E6}L_cd4(U zItD*+SPbn`t9bZP;sBNx?dP9FLRm04xMj|@VTL8ALmSSy#y;iD&*+$f?aOidI}Jsl z0xFay3M>3w%LVc-Lne`gp~_Fc(aV?(UJ>H@^AiUju5^bL)zl;m3=E(~E@mr#cXf7V zFtOLsA$v!?Kc??29~C@*Kdytu&RXIz+@lflBmiu~2{aNXY6I9Drk!F-)bN(@e-LVfNm zPa)-k`>{hQ8nkU9h~sR8^ZK4P>Z;wnsnfjj=Z?g6_`PjswRXz#|3$F&M2Nz3DGE=f zHfJ!;m9QbaPuM3O>2v(P^7h<_!MLuqPlP|O{a{Q^bK%YE(uuXNcB$LLKlNPS-S?pi zw$F@>7k~EFKdhNomvfgyt25W$LN_t*vZRXng(s}LaxP8Kk}YQ8>YQE6O{l7IBmR9McR)gt;h7dkNK<^0I%KRE$LY< zhs}$&V71jIR2X4caxIP>MA$o{r1x&n$3kiVc7S%NzjdzLsaSDOauF$+7zuONP9NIo zlM-}g5JE}B#Dv)iS04<1uHfB@$ZXra&5K%yv*S~J?)M8?F$4$VyokiFBsPFy`okza zh8d-u_mqJS1w`pHEaGk-v-sbmTa7V1GA|yEuZ2L_P&7b16qbnc#yIfQ?zH&%K6GK(;m9N`mMht}gYV7|1L7j!0@v?Pn<{!9B_(Lr z20fe~3}3idvuY*{@rWsK$o^?zy0Ag`oqQaAT9Vd9 z2yWMn_)BygI)s&5G+mh$ayJzWg~|yFg{ABDH9XNb^o!-y#y;_=?(XgkuX!L4(V~!> zn;SWFL);FRnsFQF*zSUzNcq9URZ2alh>lcl;lY0G=E?c z8|9L|&bD4dC1G$G`K%kEKP8LPX@i z*d-)c7&WXTZGLUYwCL^IB~2W7GTM)4ZeEm?ld?}@0|}->L+Fn1|J8*IJGk+cwbmz2^w*sDR*-3hFY3EYFBr&7mlj5QOV%%1KB{j{ds*$tWnOGa2CN%u$z(X6*BF ziUtLqA?OFl69i^z@|dh+=28lUFb8WT6W8(j#&$v8_6X-tX2~dtbx7ixADHKD*6~M2 zR3yplM8^tfi_->wEkM1aJvMpW(3Te;t#OdE+T^TvklWWmi+=XjFwZ{s4TQFvtRIz7 zjc0V+E>GD8z#if^W|HG~y7AU9FlBIoOKdV(bGa5WDK<-lh)f=^&$}*WC-cyr4)N;Y z%i-`BHcpJ>jIX2!J`VqOdQe4F+UIG*ZJq5#chJn#tJ9&;_9uCV7Heg^U?Z0CzR^|) zcS|qq2*!3*xIgv%_4LPLvj(G|7#xkoD)E+@y1Me~*Ey16*1Eg{yH}<4m8|`hc;EXU zb*Whsh+YU+GD*jEX_1ta6YtSvX!5+0HpB`qfh8xYzD83jydWrfOaLV>tHVhu&p!@jinL6D1XujJc`AtD^42C26PeNaB?f)ic#teTX|=I7 zt~~?d%+s#yE1Gl%Aw=5Xl3m@OR3r|5zk=S9%P?!1EWEc7>!89*=J$-yU52WK`6{5G z-)_bFxQ5CHfK&?RU^ti^G%!4@Znf8q(U6@T0Q%$=W@U-u^@0%Z(IA>xhiLD&5Pxp8 zNS)ZMGSejgqd>%ZdB|6Az<`-CkjQGSe*aMGQ}|F)-?Bhd4j6GxhUkYu5>*wjq(r#+ zV1)AJVqxva4Cbnxw`zTL5U4#WboP+91T$1Om|ac*HCxI~n%%S?V2?di;vR(F%%7PQ6?IlZA+X{O@&^Tt{HB3v=MTzQg&I8xDJzmo*dOb z7HNC?>W4|ngo3pQ8ZTmrwuxiJ}i?cYOzVptarkmie6Hrz$dA=IN^N?U3|Kn z`9Nd~d7Jim*cOIh()R}B6xA=B$g4yKctp?_a9;QYzm9^sYc^tE1rw65e(Zxn+F1lD z-xaFM;~|D(!tW|LKAWoGqDA$715n;^0b3SojZ4k9>>Ta_ReIGfanwxwV8>%zYWw}* zyBPhl9rLZqdgfAUdkg?J6ujbc5a>0JO+K0-MzJb*lCIm?*pyPc9(5s^uqCu>R=jlt zD3r@)j_dgJRIS*sCN?fETQC^U8+!`>>E_l}f!>%XvXKA;(qU`nJnxDllm83?8PK>+ zg6}+u5<63NveH%OwxOizCqmRgf&XPX#jBy{;e$;yL6#f1b8inVO;*mDWOQ!nrJSux z!em>B5hdbSa_lxClxW!49?c}7bSulrt1svm%+<6${-*lE}2_GTe@35H@2} z9l;GVfaFc?pd(B+JoYuVrG(EDY+ne`WWCTVIpvYr+S(ewy}2$bER=5bUO~ddKjEifWmdb2xDEs>G$vFYUZAO+q%unFt@GM1B_pv#jLN0IT0K}3 zUeI;yJ98r|b>>qr32VU}V-zz8$Mx^3k3-Wh^d8);{s1?ey_3JDIJNu zzxW|WGNGw>bzX4QV_s#88>M4oI(u0&=1H!^SOCI$P(}(0{n6IfNS;cxx-FOE}<73CX-qCR`JPB;H;EG2< zr*(0-^0Klj8;W_!Pah|(t3r0-hK-hMHecfh;w^dr&bx2sumwGa1H*IGQa6X(@f zp2+7lVFnzAe59zrym(C=uQI2I-$cEtA`wAH-YJQyqDT0Swrr7;ssKtXMz+C0C0hW* zyX1Kq&IM}qHayCQCtGF@q8CTEpYLb9wDb&B&Rs@KA@-mMxulN`ORaxLJ$+qqCF)F=v$rqaJ%agm zi=BxgF3%HlwYKxA4RbqG@6Qs4k>yU-k^Yz&ABbQ7hMLc){8=A_ejdBuoneiR@GJ&g z>1Q#G<8|REFA)<+!xOY>WKdTE;FNcWm0|f)%+eVTUh+>1Mi%uI{6jf}b;kUcp^r?( z=BN)z30vJOy9V*2NeEPV0PxD;WnhqQH4aFVpDI5tqf5ucuo|23>h8IX-mtRzpm!HCXPYilE#SIOq^78Y?Ku&B@Y8y-#KHY<_wR`z8N*R9b1L6p|yRc|r9%4!HAw21p zQ731#?=Yy%!5+#GN`0h|QIf%V;t_OOzY=xYMZHQ_7g-^HuFoBDU+!(iaH5Y`#|kdn zc;|x9r*rDC-3sTvZM@ieq=NR(&v-=(gXpC)%Mq0PjSY)K&dW`{xnwZt$`#n_s8Vn8+pUE4^ zt+}n58OyqmyeX*Cztf}~= zY8`x^N?t2uuZLH^SE#rWBHQT9=z?GG!N@Q>+}oEyYd^ohP4A{G;)FN|GlaE0TBxyieuopXFk6t~Qj^OQSSAO&>VkxBm=;kPW3BPI^h(=Z z2$efoVBf0(f%bP24~7$8iBC23wSA<9YMrY|wW5!e+X8z8Gwt$t0hdR-c!?Qf0$vns zcwO&uhCgz?+y7qFu#SO1gAXrKuJ|bA6=Tbqt*_xOwriZ;FCrh$L$oIAxnn%OvRmhe ztYFY?c2xk!W=Fui$QdA(LDQDY?Mf=o&HcRIT`(^$1aTHSL`Ft_yy^y|Ujt`iNu6&< z5E`;D=RRXRoPyYmBKPT@+>0cy!9l`_{%E_1V*L?Q4|;rjtV&*}5OdGR|Bf=D6Ovom z%TB8swjWaxh7#b$v-v5sT;PeTJ3m`gX&`3(nHoj!zVtW%i1rAGxS+2tWt{TrJq6-l zmDSZjS~EZED{E71s0NC1R-Sqxnkl`|MMjB!vPPMMeSCzSTakqrd8m;8y(6Lod%w!d z+<-J(S^YJW#81c%T3bL z1xJ`N40OVLU-f{Nt)#>N3BKw9x`Chu)QBrSCMM>}JS{dhE8NJtB171RUl)wcI034< z(G+egb1)K7#^V>rePzN**fzLJh-{0>PeRz!RSH@!DdTui%p2{;g7^m+ zry;UpUo8Vqp`AgI_C0S1^a?BdiJ$W&j3(kk^u#K(-jyhR>*$5&!9tzFfAGLeX)`!D zxG;&W^h7dUQ<35nii9J!opmU7Gv}pwFNT8wCx>{QYp0VB5*(*UVa$G(t@zxJ{$+C3 zaOHkMnaWgmkpHK+F39^}Op1(lZi`70JhpiftQ!8hFMj2u%@r;CUp1Sn$wZO=A5CW& z7FFA|VPb%xySq!eQ-<#DZlpty?rso}M!G>lLOLX*K^mlyPU-wM&wG6T<&U#x&wXF( zyv{|f`%%k6bVa;w7j}}Lsa9z0rN`>QNo55GPJ98QuN2$ znJd{)4lKB^HZQm4o@fwP>w(fy_)ysXn*keCwKFEO|Um9T(EO&#{RU z6hWBfcwAY)Z~BtLoobv^pqR8V1QPrJMhka^!R|jt9fUC_)W(jAt@Hk8T_P%d4+u}E zfFYP;p0DAPTSmlJJ>9Zl(*RAblCwNAh0zHG_P4xyV;`JxU@CJ~b2W*J0pO{)wGE+h zm^YY%G$udeJ`6WEQet2WHz_c7{3H84)x6tLTWXn?=AO>DzVu0~L7WGc8D4U05EsOrruYG2t(vckmP(=W8Pr2&v^)fI)P}vmo13#-7gMID8{%Fdba@%-#@R$ z!Qgq2*X!gLuD9Tg8fupXrbdhBZ#_&)L?E`V;}NE0L4d2gz|p~JRczi>8{4b#b>~sE zRf_FltC*c_Rhb=~`{-d~McrQvEgo_m$QEG!U>K&484RB=gYm)&e@NJV{%~nw5B7pd zBpPV664V!PF$6BY&sm(Ke6CPG@Gk$M6n(nIt1Q8aLC&0@xl|OTPhIV%O;qu5QFft< z(gS^a-bqox(TmNL1-6rj8u=WQ#N36?!A2RDM_xDtp-INzY7j-R;K{s(8Wo z5^)tLkO&l3Hv-6|a25dT)4o8-nw3YmvWn^!k?Cs5aZ;;lP8Sbvv`K$Vnnhlu+NwRH z$wnT{qhQCv*ziRA*zho8Cd%L&(vwdOKxg?9?@-;TO}n7&#LFIA9S~tA!h7z{;bL$C zJkR<|gKI{;l1)}&mdtmhHM+P?#Tes2_6Qxql*_y@m13@;pu|mQFFQKpLH9a-e`hGQ z*mSmz>fT8HNdM<^^U}$I3&6;#!JpBDea0o^r@=?M9b&mQOA2Tsj$7=-pPIFAt!?SRZ-oZOb@BzowH za{b-Ig#es)z`0a|oZ5j<&W5G}WJaBAVOOxto1_ZD6r z1E_=qMd%v`S+YMMW62QF{qkTht}5^oJ!t0k%_$3tfQ^o|0n-SRZwP>_E~*=V9^7)h z1;X3W@#c6*qGi<$nZ>X*$&@yD5AhDUKaYx%5|TXNg_b-UFXkx0{Kntl>U{DaTaZ!y ziUmP;!4QxO3F4wSuYn&VjG(e0C;+ba!s2YJvd7I$Rv`O-(p&7slOaH`oYroH)BSjtMGVaoaA?FZh7dhPUU7UH)kqzv@jBE z_FUm@zqetv-yd;hJCd8!C(7O>L;RGer`O6#P`1x&T|<=hxFJO8MEEzHWSs$M0Hj=0 zueg(dm{$tZx1_=9nY@}-&7fQa8~#;Y!+;jfM@HRnN|7$oQv>03P3n&JBPo|Iw8(s^ zkk(TRyV zY>tBG#K+X^sMwWe+X6%q9mQry!1sFN7Fnt;?{9QEclk$*VrA^U2Vq~21B-@Fh9IoM z=lvP)qO`BIBA0L?I`_owgHUsIMJfIMl)=}y(@Y;2FyHbk0Qvo3?_8QyYA$Hl_64@F zL?cM2`0_XEP4w$%!w?QZDI&2}m7|h^r%uY@pdA3Q6Xx^qsRm@VtGz8bdyz(MGToirs8k?fF#Rs+g0n;RzadMqb*Zt4dm7t;o$LMK%Jf&frH6Ue@}LEUR8AR!&%?151CSS6By$5$Wy3A z=mu?~GB}tnK{c@%7jKM53U>@c9AUBW?ac08gO9_P5!k#`U$(-L*vvB8qCbj(?Mv93 zX}w}GZTFVui`nP>FHXN>3=y}pPt79}B?r^QKyD6;f0-QLra{f`*-6!{r8fc35HLP^ zLet^-tMO3=Rq8>hIIIp%@r$zQOw*{4=oI~DttQ%$*p^n&+=h{sx&;?r@3u`ZazwL$ zy;t+S@X~|G%=W-`5i28O zEe7EcnIVsvI1xC~r~<)TkQX|4A?O8VT{BXqbM{)Xzzk*4O(i;`g!lqC1vU= zf)($?;F~B#b13=ue$jaqn?DG{n3WV4=Q!|CZwc%2eD~n3K#!+T^4Be~S8DU|h5!7` z2E(J-lYPl2Oy0gxrO&!o5$xOkn2fL@La=h_@{h{-HpqEy=+AlYd||CbUKR?h2s-`-JCJNq8N(^kQD-fImg?K((9cThP`El*1v$dLc3>yl5K+Chd z8S^JzGu9nOf(@RI0x;j6k}(*cm#K?{B1CBi1BVa|ik`sV0D&SS4tzK=ZA8lJ`2SE< zOJ(_=;nJaRf_iX&e^mvRrlJptS6nM0FcB&}sDS)$gOr}uNc*!xBVc!;uClS72q`3bO(+u9wLl!5sJwx~R zm6?))l9)pSrc|Y{vWBk|FJgu@v%@tfN~Lzsu;|mx7Du`##DkC>uM0GrZ`=UBku%f^&bVL&xcW*V2*W+=cl8!S`#=OSVvIyCb`F7~VW}<`TkYG%$3uh7 zG$jW1T5<~l-S2&%vQ-$0fX-yke1|8&VW_KH9d|-=;ezFW_kPUzrb3e*D+k$liv{4MsNsdS`zb zS2pq|RTc8$bPmN5ZR?ICOq(gb5?o$`3m)d>N2=CjpdA3_9(TU2&V`{`t|;cgf;$Fr zGb0}dpc{i=1nP*vn_Wmm^Ilv9iY#9?j|Flhk?F`(Uw#IxYSVkvUEVR^&XsCR)FZ#! zty5c3FRf-;js9h?&5Tgr?ONv?r~hgG%N9P0w73IkOzaMDp)kH-RKx0CT?hME z#;f} zAjjy!GRgB9W}1ecVVumo%v%DiUSBR*ZmyB$v~HJ7NE;b}5%SW3s4^}-ewCP8U1mrN zTOQ-vm3vrM_h(Q}d_qF=22T=kMkfe{h-t^D{nIos;Yk~x1B?VX0w|veEO}s-3rr^R zc+1kJzlt6Qz|U`Y9tp|0_a0CC=*))%VN;qaA;yX?Mzfwqk;Gv1QC~L1k0~2eU4>zg zWiF`shyc^ufs;eM$(a~Z&nhUsHfvouKR*`%BaRcqT&u=GO*Md+Aath{Cs^xMT{IP> zH?HKyL;3y(eGy&~fN@a3i`3EAW+FAq4PPH?J}cYk)^O}D3nCua z$48yKlqzMXp1o0AX)64sZtE`ns7#9~j<)I$+PB*(E{KVTN6R1F26XoifLl$|#dBC0 z2aI&^UGFEL7*0z|Tk&uaY|HTbO{swk^$#!@6#>q!1YYMgMVnj?yxtUGj|0see*YxC z#-f6}d}=y}Y!T2Pm4F+YeI6K#GI_a7KeQm6msJ&DHR;OvoRab%y1%_vXBMh1shq=~ z%-a3rR-!)ZVfqYQi5Lw7u8PPj^0kM3fvqiw##MkjjAnD$4`@7$Y-_HEI(J@c6X&>+ zO+e~c-|deA)~y5JU{MJCD@jaKbm8-$OKYj6O<5H132W5HF$^T)*99)|$M+QNi2btwr}?}yV8LhPy!ugRy~ z-{Ov0xBS0_!ybv29<`e)3k^h$YiOf?Xe?-ofgMU~v};WTl5(y0Rr(4495R&c9PJC| zzwnOO3-vTwF8@lIn*?(rgb1zj1>#~{ja;1}HI*$=euXrgl zFfpmWzMMNiUE;q+06h#u4n*MB8ez9reQ)a?q|MpdvMGw7#l8#=O-e{uA#3JEm`MJ9 z?ErDpUr+1m3J6mJRe3d)gP`6eJSlJV(xmKbZ15Lom1K`S_eapuH2QfT&>5JF`x*0kee_W~jvR-dm zx*4Skk};&oWUcQCyB-d!VmA8xF?ap}yFE-E2G{~1=7w=+pfG6 zRH$y(>&v5-!5u@SFY`$MEe=2N*|&44fmAQU_gAF5(p+G1*Ew0_3Sq!T$dAN}4KESzTAT1TXWBotwbo-fe1 zsCS@#E|N;KCxbqOl-`$>pWSpYU=Z3<7~w&0k=NL1B3iaLdavE-o!l15On zR=?Bp@!$LNWWJ!J=rrlC0rypa{@w|hGY!>So-nR_Pw-^?F^Vxa5%-9n5gWkX}?|6QOICa%R*4# z31J(s4h4#}s}A)Mqp7{OqGoxKCXo<^$JuRe#2MDpSUp<){Xao51AAeu#<9wGO&4sUrWWMMJQ zf^Jo|ZL(I2Av7K!FZG-OWC1!4qb4tfb^9mp&n1;qRLX!C?ydVc&q8@7cKM@D_XDm7 z&}>6u2DKfsKsJ#%?*PM^7Ls*&Wj8RT(QkGjtTAfWt_XsRQ3n@Cju7tU2d)E~xo1TP zB%d8y{#gHLxt(8AC525s+C5dMr{r3s_Cn%mS>Re)nczjPQGA;JUE9%;>Gx-FMbW`` zNY%gw+e*jTvY{o6PR3|>)Lg0|!Tgav_x#c5kD;VYOn2$kC7WJ_L9gh5*>{(Tq0~Hq zSpSg8IjNJ{?JXeH+n@W+_g>cMpIWjF=CF;*sSQ-h7{_0$?Papy6&guHbI@ZIr@s;}zk{)QmK;ZKLoTq<-c);8Fs^CPvWR zt)0gB{JBp5ueM&`c6<)D2lD~i_CYgndhC&zgEMfgxOvkeY1)T*7+C_=4fm-UWDR#Je1XE)VfEG zzSMJjOYM{UAE)t36AFz|`(p`~k86y59p&d2rVKCH}k1!VJxUyy)>(#{7BqpTW-<`{jdYvSGrUyWwVu!r|#$o*5b zt98wO?3ps%s?x506w5O|md(wegG2dtOV zKnt@Bybnmjve~Dtsu$i?sw3_JJKdQ%37K{Pz`@`FEf#Un6%hQ0fP@fpr#zF+C@x4@ zf(as=*7|3w?B9oW&m)NJ$_7@NTW1Mv+ z!5$5$W_2=`I(HhWS(f(Z5#k@t#vg#6P?vPjzQ+r>#IM|1+#pQV*nvbQcruF74|uC# zPol-dd8c-Rp9s4*jvc*-;c^vqB5MA4mZx znv}8R!`sP^H*^o>blFT*gP!=G3RxIJE`q_0Np&0Eu*kE=iwj)H@b<%AgF%9XC!1Xe zFa-9;+ZMPF3(L!by5s2fojj;DF(|CHcAVE1m$Pxf#a(C!BEnV z;58U5_%KzI!{(FZ9RTF%`9+ZgxFm)NlG>#?br;|`v466r)@DG00?V>6cs?hz2|Hl9 ztJ3WRhkcik+<8uFE`t`#J2cJ6y_h-RV zwo%PH5^+Cs+ZXXWJbJf)78u+xw&AtT`+{RDk}+*`R6A=`?3C$n2aV}pcBXzH)~x`X zd9s}>fe%bA0-{&$NVf_btf5yms-N&#ay@qXJt80J)ZBh9JtZ%u)X-IC(y(rpgotkE ze;RbZ5C$q12zA#WtYNF9$vFo(GtxcSwSJX%N2e|IdCASFaP0vvJ&H8XtI(Wy$a%={ zLIp)ho$m43Hr5WlH2 zc~bQeQTcP;V4Uv;(CU*@Jr4m-4iBr|@IF(9Ov|iJisc*7i_nrW{pzAoENW41L!)*r zK{PXD^F`eDkQ84uHj#MVVkqSaFJz+Akn6{!^Y8!g`4z6ZFdmrtu%zmB1*;>!FRF}y z(S8{`N6FKyv}aQoI-t-YeYGbXg(Th)9&(*f6&_mN!1OdtA_0_kCgqElG} z?6}k&orc>E;Rt+@!u8mVyaQt}`jF(oV5uG=>Db{1uKR<7zoY(FHjg1FFQ%VAFrK~p zgh_z$QnTF^vWNce&NHan!U28m9VJFDMPNNVWzA#g&2;x>E4d!)4TU2`&}uMAGZ#nk z4hQtr{`K9I`EaVze`PCBFr@_0pr#DfCu(@1`WuHXsD zn9n{^u?RsV$@HV4^}3@R;VZQjFX7Xatiz-RDDW^$S@^C1{cxLi)GGib!p;vCyN(SD zvJj!IS)G#?yn~_M>~bv(LstTu!)GJXV6cMsK?6)rVVH(Wx0{k4s8_CIign5qikmhl zR0N0?UjfhW<#G5p-0Y0Mk}3t7WS_{TB;usIsv&joYD2nMYkVq*)+K6^Ad9Fmj3^;p z!`q(~U^UAsrtvk9*Z@*_B)g?tG%d#4)~TY5<5)&m)Pa&V{A0^G5v{^X!bX~BrDG(3zO0vX}LFF7CE z07#Iz_}fu(QD&pMxE-fA+n|0l09xn=n3Zn^Y?dUJL5H?Wpeg9DRlP*`B4@L?i%i2l zjD$h%C71fvx|9eF(*D)=x1bF0ikG2n7NM&9`E^h!ViL^cjf4D#FOm_$j@}#rQt+XQ z;p7}RgITihdT~LvveG40BLtuL)ZQohynG?u@7euJ5O;${2KXlGeCqz>jlOj=hajb2(KPvU!6LfK&kVxT9!5h4%eyMm#o3|TQoUxJ@U zPPMbDK?c@)Sp_Vh#PkTRo_wOejzp1WLGXS!+>*PkcVnChC_&3WzQpa-%D4O-lexK1 zXZ$mNqIX%(WF!Js$5sP_9}^~vU$Te)&SyKoR1si3N3aRx^%#7UKYwl;X3}NK34FO9 z++tFt3CWBM-MKUCQXU*88~DEBirm~E|E5lUg*?d zN3|17ZG=4zk=36_7Uc5ESH%rRItFcO`Wz7fkAl<3zhoFWN{#MO&V+GfcGIk5lTuxS zcND*1;z3zjOksNYJEHk_@T|`!w68Eo!P6(80p%FrJ4FQ6)#&SsY#sFQTvbD&JZ@bW zJn*r$Af){IGv}0OGzcC(nCD<}x_K3sUW+hfG&j?0NG?~1gx;1)yG2CvH;;#}4+~A5 zwD^2T&lDAu5J;EL@A-OxDPtLsJ+P1l*_=`*3KtIRgCc&Lf-r|7LH)jJz4~jOM z;4|P6)6C1M3k#x+*Z$qdwMg)@254@acHSEMX9*59a48Z$ydbDd0}Aex2g2vqKW71D zKJzAQn*~G@fL%@&h^py;Y$C7_Kwwgog-*EVxI@Ckqienms9xJpEcZ4tACnzsk5&}Z zg>oviod32b#%?->9G%MiDzH&w_l3frFIGdW3i+*|-8k||D5YwbThN75DWM}Mq@yUa zN2J~;qFblDl6lb+MXSXsMW+ls+?)RQ&i;U=re{=w)=BC2!pZW^qO`*h8-~w(-FENi zcyX(`dhGez&5^OEOJ--Ld$P>QZ$uL;yEo-x-Z=py`sG8|)GUrapz*vbn&scSs^%jL zLJPTb&u0pIzx3apZE}Z+BEIflO^onX=T~7)C2dO3K*(qahOUouDTqDEm%tTYgs{bN zeHzVjNhZ$cGPWZ8T|ssJt9QuOcl15SZclkdf4l!ay93GbQQn5h~p-{)@=X?tq`Ylemm!}bF{Q#$U0Eh+Y zL39Ryllj5(8Mq}+2g4$yuCw8E2|^fAIB$RK60^rTI4`pp;640c*koa(^}urjm+{{t+v(!ldp->?_Nf>Dcj+Stl^PWVaF zAaB-!CpF7Jp-zUK$y`LlZ%Lgp2vOJB?zmi48xxwW$X0#Uyg4*O3h28!%(3Ssv_7gE zkCRn8j$3GOH&dC1*RvTwf>3y4sHPH>|D3opechT7MEf)ha#oPRUQuafJOF`h-k-70 zMM{KMA=0>5N#tE3);f%6KySNr%j6~6Kw(dD;Oq!xh4`Cm^$3j-RoV@gHIyn65qGD= zjP_SlyTzS~e>*ywh~qJvwl@=C04gRufWYEvglUBk8l5#4Xij{FFBbTnYZSlOcg-sVbs>2dWq8+_Voc zE1_sPux_v*CQV=PIcNq7Sg-B={qhkMY?{blQ&ZEFL`@;Mi5m0oI603Z8v0H2oeu zFgdA$c|<@jtb1biJH5F$j$+CLbbeOdQvAx&vyNZ9>3{o2#(tqz3wa{6`}{5cd0SiY zz-L!|^cKD-WpT^Fg~M`GV}s{x^1LIku&^}eRRSM0aH}r?7jOZ-u@{8pt|j0XZ8`>c zXTjK4=^iDN0&w#r)LJjj*Tq0Bbpq_y9Gg2xe$V&xMV}wse)iT2VB{kw>6hi6?hzn z?r%p@M0|-gT?9;6k(ndWd3B~n;^F?B5HxrB`hb<>mZk1_9`8GVjrWRwx``J*!31xI z_**c{yw7Y9X>k?u2ABX%%n8zJyc_dZllPPCO<|3Wl9r|ohxij0rti0j=rWkI-1zVQ zm>rVL0SbSwxCHBgX?!8ygk9)tFhgiNqP>KTmByR@PN|~gY7ZTKCH1ab;j`^U^#{NX z2E1vIpt(r<$e(|5_LbCj!=h<}%@8^mCYEVe*v#Ev-RJFDwhzf+lxmt3;oldB3}2rq zK9=0p%L6^xK8jF|CW77)w~rrf0U$eO7~~DxH=9R<_LIX;9MUEp4erCXC*>ZA!srk$ zSY=^l)fP<$;9=PU;X`t;IKj0U2{;ebt(df`%PkPQokbyyA(Yhk;ZYN8HjAiEU{7Q8 z%?TSMFl@Sk*hB_42LvUiRCR%`F9AB}C8r~4%yZrNi_~G^;f&W#&bUT^{3#5scQ^kg zJPWUPCl2v-(6s41*5pb25A!)DJd(we5Cq07ZXhFnSY!#b0(#d;i_s=hl953W4m<3= zif}QF^!n*qWGh8Qlp{CBRZ~YN`;z@%ui&Ka2B6_aHshefn3_MR`qtGj(Qq7^F=(#X zx|jbIcnk~l~N?f z!8we{nYI+_c-rjWQ7%n>7enS1j@g#f#wJNzkMCY}C1D5#vLsrTZgmF8Cqg-rGMDYA z(>Fv<9~5v&Su65gFMi-)3#UmVD4LOP_NSV3etmx#PypWJr3=hnIb1SmP4k_Scg%I% z5nxI(hbgPqzZZ@{${FSWQQ<(2c}I-&6jJAGe&==Eme-kmJ=9P?|G`b`hM3DE8hzR^ zm9l2dNum}%^Nv3bi)wKV?sJDIWgTDR=ez)+@zUe*J^CqJ%m0=c zE@5avabjCSAQ0$a#pO<4zPHLqVPPRo^ioP?Q+avRe=US=3PA|imbThsp#W{r-5z5& z<1;E2?oN_6&I)yM&C$Li`85Fk*b!Qu&(|2>3K*K0I8%i|6hrx^f3ie5`C589J04iH zDQ!f;-RjN8!VT5Mk%{PgS8ox>@@-H9p`#|~JlJ~qyY2HKnXD%8Jd~@Ksqp!h-Dn52 zCI*8CIX}+;20!IDc`&5YQ3>xA(3=*;#2{q>KTwQ;e#Br!v9KFyc;b~s4~F0gQ;*Zx zdI$7PT*$yNuHhJPcU%0nu)v&gww_1w^8_P}hm7{8Pd0{Yt@>5pgIS=mK+Rfu&)l^l z{894^*Pfb{mazZpXZYp}7iv$sv;uk7+V_%LN<*0~9`tump66!>>&to-inF|xKNNn9 zSv@c2^$l_9E@=SeLsjEz(bH`@CUefkNv>3L7;6TVZPRYu*s_ajn@rf@gRWtHjD0p3 znSb_xlLx4w7ZFd!pE-M|8)LZ0gLGlcc~~)r^`NQRCEkQ95-6?qd^R_Gsf<{)&kGbq zeAGzfSK@=fSJk0h;RH+3K(Uo0}#@;biL$KN|g%Z zhib=?f+`NVsGn{HQb9oY2>$Qk6WZBO-h3KRbR6%dcl0H{Ck57F7!|=MOjW^!xjDuV z0!x+zNI{$^MR3qO*)NI?2-Yoq@Ww#KRG#s+Uk!}3udbL_s7nICRPH|fj!4A?ZWECF zZ{Ur9_#iimFxN?GjQ6WwO`U@`f?CljTUTA^lX`EeiK)c@Wm?fPRn=*6IxTy{vE&|2 zK$gW3^ZIzkX-Q;R)5I5<3Y&^D-*7G-2~2YIvgqW!@~6X6m8V zZ}&cz*?fn_-jd0}aKm{@#!cq}ACE2)S|~7;+a%WSgg^_tlo4RycH?7EYMt%4a+@qC zKD2Fb(P-qoHT}f+V4i(1&qP(4LBYWQ^s^Z(u6`&$3Db<8?}jW)`9ZyB7bAnf1IWa|&{tnRYX>+lkbD$vJ(k*JfEkt;(^~_fR z>V*;#ZU@JNP29{J@{BgfReZnGeODK4JPRrg0s~KFRNg6AuE4)cE$AxR(weAhFIh8H z2#owZ61}wg%p2lJJQ&Hg;nQ(YC1R|ptL;!}p}wWuw$IJi?3vN?1#S2Hfs5dYti`LD zHvG}|dsCXZ)|%p|ByqYRSnt1b2q{wmBQ1+7Tv0VtS&V>B$$H3GX{gFmL(P$il;J+Z zlrfp0IqIpi-!^c1NK z6Nedj_n-OuLC%C`Bct^ZOBbdzO`goQPlL1Iwln!h@_i*Zz)}PZHO5~5nk@mZ8Up3F zKvDV1sM_%;a_XgW(-4J~ur`*O-QxVpzScHwy!x{g9eWQ_YPOe^H)@a*=;rug z>cks$q?_oXyQzC_R(IC$x@Us`^6(p;@_bToe2aSky17_dCt5k8$PKmjM4uo(~B;++DB#lk2|3_ zXMt&cbq@y`xe1#9IJvnwus)#A_r4>_DlFD0fbdj~lAY@YsjuPaPTq=EZD4xhF)JU7 zEIN$c_{&vDy?HyE+}Ga*-bSTt*GtU5R}?Zt;S045Ap61{cbit|zsp@o!dOSdNaNr} zj4=8-um3EP*HDBs5X@tj3pVzgpZ*LvTPiCjP=##jBzVEy)#>Z$ zRYaNl8N!*xtg+D>8>Kn5HvLZfLU z(A!h!kT112$vgF_>8OyNaLf#$8~wfpmOYOWg4%8TVeFEpLs<7}H@>Q_Kih~$L1QJ7wEMyD&~ip|(?CLX2G3zr1s38n*sor2Epl2)G&plRD6v5Q*G4xneUm_=%BO@QxA2jEB z$9sX^B)NdDv@Zf9;u4Si!-X$cN@!S*!OiUK3ILcY)-_3v&OVBpT>M-|NbT82ScYfJ z&Cafx9aE*3U_n7x%%{7X=gXGcOv?K{b};OoiF0wi z?9th;ZaLitE;u;C?J+zLc>TsG@JY>R)VxA=!u6zf6tm1P=P z3LYoHQBuSVp|)cg>XoD0x+;i{+a)M=lnkHp$@-9qFI6P#3Q_QKLgAigxUC-j@I#=Usn5(fFvnMC|cUXpC40=6&<{son2)@4xFqr_{G;iO)I2@4rmO#+{ z;OQD(Dm+)T!Eo`R?iaI9fj_J4Sxe&lR`zBUtaJN~JIgd2H3+Rz9nBxXbf(d`eaCae zUV>FDPZFvi2o6=ik_A@wfmpWleD!R5A)ovQ(054?(CBI~qY;G+K~j94c?d2NYfi zIbG&gOG#C{TKAKR#750AL1XqApe6^j}j8)9J(YabOF=)B~u z)^Ssy^bei4veIqi-#mf(lCi8?G_;dD=UuWmx1 z0z)GPQz5@0|48M1yzlhO_-mPV%e4>cINn%~!1{IEu|V4hPqoLmcSFOcNK8AKpYz@WZP#TF|iF zeyHH52~AVkOo;GhpbP0dstms<_!d*NYd+(V&xtl26nfy6)|~c^4idH2x|g`4Nje>@*1W_U>{t%U9{44>)TI|aKPNMJw*hm_}|qJ`jg+f zw0obmwBBwSN4X*UpY@9@-Sm40O1HP-7qQ(*k^`N5I>4mtd@l9<2%Q!hVf|s3@P0iP z-;ia`(x!r-Lr!}(y=}`|42EWdcB2bk1O4K1)3pOp6(fP~BF<^@BlIk}1Pbu~^^smR z$$)Co;9n#pc9VY6Z?uK;9V@+CjkA7(;K6>Hhe)5AVIiYehwfgq6=188 zC1606<{s*2fJID#LM+A(R#F)4kNCJ=#5X)JzkL~Ult{eZ^{C=Gq7kVhCzGy^D%4k6 zlGYYv&>7R6u9@%1b4q2WFkg|h%3wqPmG{NJeuD(0xBAuOqe7nF;j0nS{M6bH$$-lv zqlV^t!lm!khWf0#zaU0CryCb1y~3YtT>qi(xKwq~%u2P_nUWgH_B0U0j(MP)eEHZ0 zcv(qJPJOlUvxUkQ0MXcoQ}|rWD-zt2IT!&xmGG^y9mC1rKF+~A>=gl#e3@NKyqufZ zw-CUiUSnW$`b^N+d8Wko5{?sQGb8#CA*%bnFD)%i8Y8cE@nMc2m?ibWhRbTOIeZLAL;Purk~c#S-a;ED&s zxhx)(?;vH<&FMwaAMvoc1`Ttpp>iP69P@+A+w}V`C0)N1s5*=ijhaN*Yw8_SXn-EX zhZjAjoY+64r2Y>^%iC7kqS(Kt)xWPX9hXqQ%q%uLuRE>U_FNKKQM=w+4De%7Kx4_m z!H~Dxbow24C$_e=-R=`D`Of0*3YLGfO*vXXmo-?=ub&WE=#-JaJhHSJXHhq)S3~+Y zy6e7BG{ftzsoe6Ok*9>YIR7qgR!e1MF5$rx*o?Z)Fr*y&bTw_Nxj6B!mN2O%)}%&z z#CO;4Ry1>rEG_w0KqdiU%JeZMAt7l}f!ei*@2N3iHVFsvmZRa5x}H{}#K_zTp_zTZ z(cqt-6$vuCif0ywGv=#z;8o-c;{FLgxu(rcQQ<$G!g<>l*rBgi zJV(2w*E|=;@GDfoICL9oVYU?MeS4fDZORaSkqHa=xV}dV>(V|Md6a})nqV3VRkc}) z?)ptwX@ZEr%^<6~491cl;gcWG|NJ&d$sg@y-3Uhg(l&1eszqOKw5`R*=XGnrbaS5zU~ap%oGq?dcRq@7wj8IKGqe&w>j+EV#Cn}v#aeba zx=@u-b8Qv%2wtT>5#dMiLXxEUxW0bTFy}VJ6TGg$81&hQCb*co=YB@cQ=YR{!f$!B zRU>a0x_v#$?12b)g@_#^&ZfRg*Kv6HFlBx>Rq~p$`g!Pk!47U13mmidvdT}PTSw77 zoA2)39TS`r{x-(#iIE&T!L)36MJuhYMGlK~l$CI`EQuH=FBkF0oAz9-$gTE zL|^lbXKnc`gOh}Et=N{u1*^YMlJx%pNm@qrV|1$F|L^_970kCfU+wnc%j%=Z>%f9-oSEnh*UBVud zs!mT20sAa_R_pa1m%nqLOC5>>o2x|_6w*Z*Sd4bnHNLCk$IlHw=AEIJQp zRmd@jAKzE88L*YH?hW^H8C@PP8J}PUq}cqD_OJ@pKkKStROcfE_88pcqK(;IqCK*( zO|56G+npzMpFQ32FN`jWu2iZfUCPFt*Jqjp z0ik}^1(OPDnA%~=FXU&N3o*kz)+r0~oY|b(E>eWxzB<_mia05|oDkMaC47fABT0B~ zwKb%c#)Fhb(v|n$C>+AA-##ers6iJ#*ZnEV#@zRTqOBXbCspHiy5A z|1rhf3>P%|_nJBB>(dc?_&z7VH(fb?NpXa0@Vr8p;d6yF(9e-t(xd>oWe?w%`=_Vuk0{UV6dP7PYFsG_P4;#cEId0RhH=BA+y>{#&O<{SMxz zk?AUhUCotmyYiEmO<1g{hlRKe1up~Ez3n$ITr)FW3h)hr*8{l|TdGGJ_0iBjW_?xl zeG@CzxBic&uMCT-{l11_fT0JFMjAmHBn9bG5JZq}B%~XpyHk*m?(S|Fx&=w;lI|M% zJv_hvdtF|9@B=fPbKiTfz4zK{aTT#@P-79Ktw*sk)ZkOS0#hcw4m8Cd!)Y zT?e%+Ye7>-O9#-`W{z?cLTVuAxIopl^eIsp^Qwf%Zf@0uc&6**C1%wE zN(UEjWCj2!0K{>^E+X(+kauR_vIMDpzHN8V3)8}YjB3t}D1CVJ^V7M;ci&`U^E5$T zfl`d7v_${J<3E9m*4#6&mq@Jk#T#hfjxCg;v5~o=k_m59Gg8tj5?XHhA|*Qac+ceK z)3dC&3NZ$puv6n6Pvw3T9KA$8SK*kkxDPyfu&=jhxWLr=%tv4`W!R`u7L2haxsWqi zg0Ywt>ET2&!8iGeVS(~XtZPSeta_=RZH>xLmWc;Vjn8&z5iU_fiWG}XhdL9Kh-H`D zWr2r!s3Kyk^s{^e$8B^srm3fvv7+{PCu2zpW4hI9$r4nz6}}?|U9oKZh{wn=Ha&#o z0%TN(sJ=%>JW;eqWAQ!&w$u=#ZgEHTI&xS^-AURvoxvYC**0owiGbZ4X-1zawBG1r zf>$*q@YDts^{-FazF&N5nTp>gGaz0+3G3}0T(QfbaO>4CY=&yM2UiU2Z2H6#ag*4& z*xPH5IUV9Q+A*@K(^)+*K+Yu&P^T5E>f+)PcW9kSdkj~YAN2_D(-Wh*82V!5`IMV`*Zc&HL-k%4oe zprD}N(zYW&pUAXV6S8}2NMLy4uG_@hJ;om_2dbjY6zN!^9>SDxF`tS-zm=! z`-;v*)9ND7#u##=eSL*!@y=q2QoV~{HmY*J{{@XuJaq>a<3?>v-6_1*kl{>Aq#ek8 ze{E-cJvVDSyBR=x^$&ejf6~BFZC%2jvLjw*9ES6;N2Oin9M3ETu0Z z(d$Yg!_na9X>6EgX*&Kyj%)lZ&WU-Tix_PRVVbgs{)i+k-8n`r(&P9qbq$5yt9fpE zNilsxWs`h=sh7GFjo(k#F}<*v`Vq&ZdK~AF=00t60z~MGq?trkqVW}aQ_cF8=1*## z?-ORDhyUhl3%Gfyv&OIwS1I!~oh{z&Am6~>ZIw653wO2w&p-X0``49rPm_GCqd~0( z>z@|oP134mnp?k|;uK2LJDil>Vg(yf`+}h-Ofa+^PzoI)Dk3@zn|M)29i+>!r>$ab z@V&+vPnI6!y3-eg%F1HN`}VDQsonXxQ*)WH88tjK0jeS+E$z2EfWfS*)knwwDCRX4 z&1HsjN!vjn^HRgKb^m~GPg=C?`*%cKy3ZY)(h)?vW|R+|?R+A84Tf)SBkc=U5+vNc zj6d<6WefS0iQtjNg+4vwl1>&hdAL0Mt#g%PrRT#oYqh6NU0C<@KZk@b4BUdH-tNJg zTbA2@CQD}O!^39C8Ze}qe%c$KC2blL71i0+M=0(&Icpy?2+)7dpMg#H%6)BW{*e@~ zv`hRpZ~ee}REsfuI?*;a(?!Gy?KFZ-+XaEhkszTW5xCU^{?WrDJnL33}7@_a0VE> z{YvHy>27k}$qN1oh;W!h`nDW3%K^@Lqiyf}PizZ;SfPa0h-|P&t>nwu4Z_c)YVGvV z@C*@6($5n4Nh0e5mPG(lxxxxq>r2#}epy(QOifSkhV9Gcm6nc#lz551I@4`+oq&`; z^*N0lfcE#lKXQbIO!>~))-2g5S+zw)gPZ;SWc9nCGxrLk0TB)k{nEg@zsmEA^5A zA+>?6;Y2JlzyNCp-<9h>v{?#~yyLmvoWSlvkuy-o?6xN+>=`6KiCGp|ET`FL`=;iC z1@%r;74G+N#?gsqZj+f(JSl9BG2_4(y2Pm)K5>n%#YvA3Yey+!(#%e1Dft2wHe!j8 zJKLXflFSL~TPvrP?njMm3B_2tn-Yh&zSvv071$F||L8gTj)A>EzWSAe6QaOq!4dNZ z40AdWwf%{w;xBa8j0>3$0xUY+rjc5qy~g`>o{gbq@qbSVJ`1*fkXmy%yNtVG+syl0 z5LEr~vu>^W1`-YO?b*Er<&X0+hP;v#FGHQqi_E+{DGK)k&KqDi%iw5wR<}fT9t-QA zFh&*}aHsnxbx>_Q{1tGM`mB-s%d4(!;~ysgOD!RdTdQ+ZgPOAX!r(>4qVu9qtjAL7OT>{GAEZ+WJH5&zzv!kNB?92m736) z3ez2_M;|`=&VDoG*)Cf;V7cA1r7E>o+-?GFVB>j3Omj6~W4Z;jc3c4J{0~@A;&x<;HzyYfI&Une} zk2<^5;zDL&7BU3b495Yy7j)QF&uJW3mg*JQyB!L!#Am~ZKI&#k)i&%@?= zLc$8Sw|0K&4gGel;7+tswpg@?taK}WK>N1)m;Pp+ep1~l4S3a)Qj6H{?|$0}EI(#m zU^x!}%Wg4yFoMtN3v^|k3@SZ3ESGuvdCgywH%$ck1&bcNS|Z$ohu;Moo7Kt`PthYw z?yQ}9S>?zMou_<(v>Wzpb(i53W~^!x1jvoj*!S+a5GV9APcgyG`p0vfRtr^}S!1(X z(T!L!J9?op7%HNK0*5e+O-mviBzmLz?@6GG%gG7anhU6M>5obT5;!tp3>$lo^ z0yf1g?C$f!BW(oNyQmbdqV0fp92v$sDy0X-o{0lQp+k&g zC}}Q#|Bl(kkhDK|^WUXqLEE9Xo>{y2iP&{jhB`iYS;FkUUOCpP81{R@Tbk|aAn|G~ zuDa1N=f$?_U|@Nhee*e5-9M|iAwQG(Pfs*8jaqiPb*C%tt%n9pb0zKWcNy`qau0p8 zwnn|Ym&XfCR?UrM0h^P4nuF7K@mPp!H>tOzk9kEdj)t=5dhu0*xEd;?P&YDW-japT zHv9bZ_f0~TYUqD4j1BGORc98>O9O_Tx*C7-2&%XWmlO6SZuU&ex0Lby6aTs- zUM5wl>@CR*V1~Gc%+dXbz^{#$6ogQYIc4mDWP$JbfRF1MR2Hm|2cqeqNFG6;g&L3xeCcicPa3HYVwfVhv z%~J9DG$RzlM&x%v@}}MF%Tk--qt!X`Lzxl(c6rV!Pb_l5KI=31wctSiPlkB4#8AiR z*#bp_@A5j>!hbni+i|XNcFV8GJ<~{swZR6gq?n2sJ9w6QH}lDv9IbZWTD2Dnqn&k9 zv5qKL`-$%G@r{^JC}ZT_1y6nS@7s664=>Ys$U%agdi%kE6gy9A;c=5g{G$dbS%uC`vgnEAA+jjtwry&}sfvsay{;oA}| z{=+{Cgjk@{`tuaIHT$256VC)))7}Q3prDL`vk)WDo;BCF6wCnpxd12Xaa$Wiy)oJi zPkX0InW)-F`Te0EYf5d2<|3=0ocg_NK>q}#N$*f3i^e7J8^!|(={_gy&7$IkV-ryZ z2I0`}m+q8(Hu#H*zu_tlH2I>a0U`A0v~kNdA`1W>~MH z#-W%hMTl&ay&T`H%@n#d*zXmt-`%Ty5bfYph)+`(sJEi(F!l!kuxM*s~_Tg_rrpxd8A@rm?Wl1?F@yJM%WlKloa5wC(=7bCkZDI&X_MH)-5kp-_7Su&ey?` zzwok#CyE}M^M1pox&{2o=P-y=T{**(#cZ#^F3OGDFwYl@f@{~vCY$UR$0IH$yY`L`z<@BZlZ7-l~wM3`#Ez}bbFk!3h`#Gx1v+{z-BJikCGLDzo z85YtxW;$v2nAJypdFZvI*4es17Wn4Qg6U3OEO-so_s*AC_b{Kp3JJ`5BOfcLCQZzW z3h%5JL@GmrSN)MGArp3pm}?>zjj!xegR9K>M+MSFMKEwRLNkGC-mO4Zvf6U^lMp#CT8s}~Wp=>>_{Fvto$Q-Iqy6gd zho-r2<2Z#0?ouPj=L-Um{MrK=Pe%zyk4n2cWIwfL)Z`>C?^%MVQ6#>; zn|{o#nrmBhQLml~YmoFrG{UWaTvL!jQNyzpiKh*hn<)`>_iqlFD<^DWR?%kVxw&ur z_jUWdl5-$;ah=tMYDDab zQq7{;8f!Eg_UrcJxJGH1BAw8}wFMVZ*3kYwX~!x7CxNUFpi*ChFWSnD@~mMtDqXGW zZb59r6Ek|GxNaWyQ;lV!85Mdz%ht;W+M)v`4I&Uhm1H*^RAXR9NI!|I z3E#FSZJ&!_@&h1D66ceG(4$~kSvWX4%9AmXuSN6n<}heuS5i5b8fPz9Dtc5zhs1@w zJK8d>er3S-VRujR*I~_rq#h0OLeQb$&|JF=lfUC;w>ob2RF>Wq6q`Ej3az_M6}}5OCD{cenf-iCVm^Gnm4G$P1ezV#>-&)ui8#}&PJHCp z56s?-A^_Fs{l}({@J6nOa9h0z~mm;88iLoQCIlnJKK2{ z6x5aWV)c^FROatZQcUFRJT#SFIk{FlNzPIavxHLvlf@R1@P#s9s#lb;<{PV^^XVu; zLmgY(sr{C`e!X1nMHF{>^h%BJ=g-o8>>AaQ&x(pQNrwQ*{?+zaTHK7P6GOIjhCYwB zjt(op0t>bT_>{Wg(5$_dz{A_<@WIFCql(K0fDk?u00dyQv z2-yV$AIhjKH)zDFZ^{b86nk!vwGs(F*pXwzPHD0KjZ&?!T8MIZb)(dW-ufpJ|8PNA zgAF>I{ZmW;1uqH5_vPEX4C)&QxYy#4l#s$9bejT^ykjIA{W?m>Xm4*-vE7Zz#Gh?w zJT@f~qRl#6S{kC!HWr2=)4ciNSn1nK{_KSM!0PNX)%is@*Xywg?s+QsQPT;Mz|3uC z?PWR_VOx{NcP1LWFT6}o<0se5>i>!{f;|ZA9BRVy7#lRpt=!7cd-p%4nH`w-nh1DZ z8hqbdbg5J9cUo(X`|-2n@u=U4lk4>-kzRt&8jF^a`f&UpV(lM#IY-p!k(u`wH}2EN z$IABQ=ifGF4C`+Bg7Qv*ptUJkaODjz1S;~^%of%YaAOmXH@Ja(yV+ip9j5++O^qh|JdJx~97hMN-rimlb4CgfbA zB!HJ9L3Mw{`-co2@kb{N3Uov`v^7d_5brn)<3Mo?hqxl|f$A$6K~hmjUTmNzY)QS* z7-)R-4WSs8W$d8o0@mgB8T|+Cbt~Dio5{+f0D&A3l|7j!j z=u9D15#R*7LM5Lj($xm)XT@sKlz@8dtAt7r)xl`Auz~gt-;kk%7PQ^KH=sIpC=-Vf zgOk`({2{jItSKsO>$wO_>BNjQ_N2DbRb}HVQPgh?WMmFop+%GY>$4Dy=G66xQR-%E zj|tZ9h^cLQ?jWb)GfiZ+SDdBTyRlYFEvLhUa>MC^A{j)0DS+%fYz4!-xA0F||TKJrWioq~np%!|CscWeoFiV<{I=kAN zbZ)9{zTz!n`@GV*_qZO`oZC3*F4?>Ns^)hpYq%n$ zo2~8dgA4%&=_rbK=DPec6`=ype^&G}jpREf|N2)NXosHOAh-Dyy%<=CsPjfhip~sd znE;m6H-L!l)ULl``EJ025&OcBS@kN44kE!QMG98*$hM@jBU)$t>l0&+Y%mqgpF7GU z3h(u&7N)+)X5S&&s|8gySl8+SHLRp9(DXegV_1)&%z%1#p}^R{W)Xa!NiL8NUOA$< zU7&49QYydGXFAWSp$@NCp&S>#%%B+;k2P)s;M&9M1`oT_czWBqDYelDsni;rnkBVu zLGwec%-L;nBYGkJzc~&wl|HFhaa`PgOE8))cP@G4Se^Oamua3776-DG7eL)|khSj& z;2d0U85yR;T^ubx?eFu7a|h_vMj|k8tyKxfC6tJBvbYXo5RrNJsZ+82fLbmD+b*fM zYoI#}8DHG{+zJa*jYlmO(75Ckp@y&N?1@kjx#R2@I)!(G4YU;Bdar9sHWQ6IFCrl! z*>4RIG&%16DAjFy3!fyt)F&LfIp2=dTN@iwIL9Nr27G3KxCFtq-KXY`!H8YcHH3^j zpz(7)7?-@3@4U-NAnJS}oP6*WttSbAVumVAe6fciKrv8+Ev6{53`) z*`=siJ0b> zaf*;G$e*gyebeHeC7|nk=`^@I`bLM2>B|`NT{0>7;L^BxdFIOtu+-`otWXy%q-&^C zUP*3Yo95fw@xh4_5k43*i&@pO)b($Af$7J_TAWl>gL)0s>1A_CHTYrLpet5*k*OD? z3*P45zD=@AE?O%29ZnjQM!R1j?n!(+PSEjt#)m3w$!wkIBJjwHS zeERd3#jm9yRfVZ~q8&^;*BEVtqTf>`fN3-!L_|C5w#r9T^=0Zwy=(v8 zVyiF1Tx^Wy2`{l)71=JydBttNJ<<>a)#SNcgw4uEVOB9PL10fpAftr)+^;YZM56=F@h7aeK~dZFqFRy`n$QfL$OQx zmi^G5A+Ojr+3xz2k_x5vCpJOm3ka$UQJ}d-gcnTYYz|(Ts+M(Qe_XTI;F`zA+n}VDKf`BSmY7DaduEMPP#^P*vZt` z*0$6b!w+;d?a%_}#m})SKj5h6k2kzZU;tPpLWt2c*|5=!`ZCix`wiKZC2aVp|>E3!&q zae=Am(FCV|{laIo6?K~81v%LSO=N7Xn>Zy<$c_ey z*mdIpbK0gao38`inHPDj=Bs;K)$7>-Ceqbv31xUsAJg2Ui;GK&JHH8G*2iZgazO=M zLR4~#h|7`H1-0hT9XGviNe>K%F87Djl5wE|rF0A;n5r2Y2PfrAUsWY0sMPOdmC*Im zwK|g7Tgjzm-AkpSn^RjH&+M$}o(!4liz{6{B@`#%g|LV} z>?pI8<)l4C%nGyk9xqS4c;(lA6xSl}@k*obHnfW$C%K18*W1OiKLeqhM=K$bED_3A z0@2n?B8?`TXUZ2R4s1aVDYM1lDxh)oglX@3kWo~Te0}|jgyVSkn;unvEy<^YZO-H8XJBd zh}E?mJaGl=?MGxh-PDJ!5EUryujJpd^=TqQbL>`3pQ%)v(4g(qby;$%{`TjU-t~*dtFEV2 zKvXg^Lk!ZQj$jS2@l4J&pG1X%oIU}QC4dO_3H#++x@YSfZTG{vKGMN)MQREf#k9A^ z`89@=s?x)W?1{jxMT(T|dVopSFW*%(KHTNrtgn?qprst z^(uBmGXqM`KW;0vU!!_#CP9}C7;0F(H(!|U*?z`WAv$ev7~Eeb6uDyPbweGt$*5+s zvDA+UEz*#B)ACu#^inIr3(cD!(zKJ**yxe#n6B7(k0|y&sLE~s*0^N#5OYJR7z&&~ zniQ~AC@*Z()~Q`oCsrfmF;6Bnx&&rhs+I}B%;3+ZoM-UJTUVnES}yL%0>2Zljr45I z{?tTeZU!~tbK~mYTUFRyz~0CRig&wOg)Yyv*hqY`VkgWF>YjtVxLqbn&T5nEIOR*! z#h{qZUE@hgM%MI6hfX~L2voppNx}!}3I|x9xC2e5^50ZnHriC~Mu5EJqKl_P1b;2ZyMii8Lq1C7v-U7Neph4XnV z_^;$sO2y2eI*axDt-47bS22B7k9La;O%h5Rb(|6^?SuyPy{TVKMfP!Qq75?tkbaY1<8jRPQhMTUe3S2+6=&`+vLGMJ7LqoUKvgX8f6se*39Uql0xr&AYfy z>_rq^Onw{(riEF=9}C<=A)6QeY0qzQR}7&eZEYDu*qLXJ*qXaXN{iiaMKOzfdL^gz z^jfF%=ylWYr-=AXmLI))AF`Y5Q>GFt)G)R`5QH@p9nTBRyR7x^rMO1Ro0Qnxlp+$u zD-Ht9Kko6gz7PEG%mSr-@7`%k15i*i{rf+XCi|^0jT5VOKik5e1TG9_d7&{Qs0Mlwu>B!w%j9FHJ7R9&NhxkMK&WL!3`sk2!JsBC zaWL|4Y=-((5&#{v?m$qFzC3$cw2cn)iMV5L(5XC8>P;*W**Xn!uGp&767;*(qpb1s ze{JRCr_$D?pr6j5&ozgiT+P46q#gZt{ZFpmD5MiI5um>bP!aZbUxBEIWvQ^xR<9kR z*aCuC^~${0-T=hszQ&dvav^5l!_*7x`l9jGhsk$tV<8>3!H5=LS*RLXZ>8HZOor#n z3#Jv%LN2iNViU%e`cUnp4BGOyl5XYBrjE;-D|~E>kB_fZBrq~sEM`hM!>hWs@Oq|` z8{K6S>|og>kN%vr5R7TD7ym2ZpJ4znpdpw0RcfILa_+)N3H7U_?|Np6h5W1x+C4ak zf3ru5_~=Ca6-E#8cogfY6F<`arV;zXkEzF`@tebfzG)?P%?Sy zMMcb2JK-*+)gP%tx2?S%yE;5Cl?PI_6y)tQ`{_I>gSdZM);%V*v2kh57)f0sB47g# z*bw3RUers$&;KgG<=NPj!j`)KVBQ?aYw-e32aIU)5 zwYRDd1GR3IF~1x*d6*II>?*%q;lqkwS7cY$V0Tnkg*k+E6gs?hfiCk-=+XM^Cq1_> zxe^g17o?0!a`V3?mln5uDGMAQ+*+o!&-Y9O=KKj!^>Oyi`vyf!>eNzAJk^Dck}$WlEeys+x7Zv(&#B`SF98LNW`JE`t{7(U05eOLHCVlxo}Xu&{^rdFCJqhuCe2FK+$0muyW3`es@;XVj_ZpPlu0;`bpjDa1 zoSg2(JL+n2{odoyuh{T15Fe2`rp+_a9y&HGr(<#W)nv3iR#Y3!mGGzXo5V)m@jls3 zIjW}PpylN={yel*nUXgnGehGV33otz-F=TRi>waXFu?jlYXj(&oO1-A^26bQwt@>5 zs1)c*3>1*H9AbnJ z*@E>C^q8)yYEhnmW!evBPZ=k~=0B5V{TdqP zXsK$3+6twm?N;W9&FBP?g1H=bvy+o`TAsMaaVR=TY^%YIjK+3ik^qIFjGW)LvlY^A zyXMED*S=Uv7Bd3SQn51nUG3Tc<=lfm56%uEuED} z)yonYV~r=Z{sAL@FL&etT?C|PR?>#n(RPqNXqNRDWq#B*)u z)<_3)#UUJF^xjOHSd`AOAmuFC=WU0Ox1BKyEVTTAQ|T&s)|8enQExRDl7l3_vS>5R z$5mgSwM_~*Dl`YP?ZF%)Ut{22%d*e~2O6zKp?d|`W%Su`p~3;c$1x+QUJl|tx_RzH z?}~Po1^*W3e>wKeJlfaK);yyle$TVftV_-CwG(Jw122RYQJcL|gL9T52Lq~Uik~`5 zF=e`6y?$=FcQ%Z1N<_m-A+)v*^5Y?_e$B|p0V=KWjPK{{nFqLOeVa1hA z1!(Y_^Xxkxa51XIz`|}9h4&}HBJubP3Q6w8p086aH@a;CjQXRJd!5#}ECzqjx4JMa zyf^fg)gJ>d`f*@w{=@u%Y$OT9jtJ4}UpwH{iV>0@m%&Lj(Tf1)3lTwN&RqZVdUbs+ ze>6|6URY~ed+&r9Nscz8zH-&KLL6Lbjva1d<82pVK^ZU);AOI5`uIQ}!L-KkTPltm z^3r^}2rOe-G#IqV;Jal-U8QIcd}Q^NKC)qKzP=xB@bqy7mc!|j1*@mk%bCBkx779h zylMY&ae@4AUwxh{yYya#fJe?+H7g=Ax|n@Un82gI_C2u~STd~Y)PRKIc$U~{g>ElJ z{Q*JQJ}jTwL4jkqFlI&dqy+6YSh*F+kNieB%s{)!8wz7q!x z%6GD&GB_yCH^@`R{B^mJ7m(ZKQh4y)!1SQ~m)qqDB)P8)s{}4q-qZsyWH5!$X=t!K zGYaq`Qwe;m1mXi5_laUw>J55LEj$kxP;#*2#VEjczscSU@g~Eg!+omezC3kjLGi*R z>=psIN98+3jo<=Rw`F5*QGYQz$GuWmODS11OX zV_3sK%n)Zay6hCE9?zF#pOCyBw@7#!Ss`oi)HPPq;Cbg<(+Eg)tkhmBm?ldeZ_u-u zfF`_cv&X>;48DQ1nY?sB@97f&@bJS#YDllqVYSe)()00&e0JZeD$W`+5|3@eZcak3 z{o`XkWeQwWk^0I35^{3xQz*YdQy7+Eab!P3{(BmVjJCG6$)zQ$y}IzB(S%$AMRzwh z@Iq&&Xl2I^J_WQYi*~E>X+N%j*rvh2l{&i62Pa z1Dw9dow>h%%_;h*Qen-6>udpPU`P~{j$N;vC6P^si>ycXK;(Sn&A&fkv5X>Qrv!#& zOTYObBkFBr84z_VavD=(A{5Cc{=j4=(bpNj#fdR697l2}4r9brc zpM;Og?agnac@112$QouV!rO1Q8Q{-}olOqgo1qNp95ghN+UnCZqQpUrnA8LyPSw#M zIHU{LFPFgC&dLX3q4PnsG;n$ka9H+R=RA|}s53T^!?4Q4fu>JTs>5jx78=Gc(Au33 zv#kOCR8Nr2oU=|3Bh#2(wSEsp7sIK=UErki%rZkQrI&8Z2;wxNlHL_kXoV2&?Z&KMLtU zZvux!3M&98YUum!9bRSDJC;{F5IAf91okmxsn)<$wc;azjgTq=5G`063_oOMW#ze3 z6$+#`&qXjG#3%%`mQZ{3Aj4Idl}CZUL-hldE+|vT)tA5K zZ!%CK2`PUbAwmKjVXWrL_hc%liUB0Qkj${r-3wN|{Cq9p)hH}BTB@<`53azpEjcW3x4N&3veif59Bp}dp3@`!q-=l@Vh|cRHGh1(UB2wK6d}zHcVb(z=;>a?5kzn)XQRc z`|NE7a1HVs>tBZV>B3<|zxHP_uV4gqp-4ZprJSFR~mckDj=fA$<--RMu|vF^>R_k-HaMS;j6GX*NfsO|QljdzR3vHQxK zaA8gFl{YLymd{BMUasGL{ffX}XW&mef0{LBBZ_`01bz?DX{Z5_vSC_6U)iG(O3n=- z{F8N}!;$`uj<)oL$FUkY7KQ)*P_a(d;7`~Jku?c3ZH^UZKW8MiR`EV|T2Pld#NjW| z*Z;S6PFLvzruqsqAGO{d&q$n^QN6jH?XWJ@0VMKhDQ|xVn@k}!R=~$x+G+Lf{?3}j zix4JL4ZxV#DGZjHox~JYoq%ZjJZn<*vT^&9GU6BMea7HTBNIiQ?Be3}c%Cudg01@8 zsig#6Nliop{xjdm?g1MS8xu$Ql+e)s-Ci_W(RXwPJM*@H7+Sky#@kl~g zmFWm6FaejPnn&xTxvZ=3Eb~~$&*0E`m>>7z&0*`Fgd4NFC%5iMvHwzfbQ}Pe7_iL2 zkDnfo&yHL=w1R?yHm|*(t|KiH9EmCWpT9;L!lJ0kmoLflFB(zcfgAnN!+?wk;YeMi zUP{C>L<1lcJ0j`8Xd>0{cr=2J^hh+N+_;~Q!(WcDj~_b6cUX$`RsRNHUsEv@rS5kdcw?XLVc#(_EHcQ$C&qN8X`$TwPoc6QRpK z4af+u%ONKW*eag3blkN(8`s96D*t@i_I{LcK3-<+k9zs$Px{(9ym`MouAvohT$cM9 z@cc^*d(G7kU_VH1ypFHE#&b6)<&z)#_}Zz*h0Y^V1swWn5l{b(+IZF2A>xh@`=>>F z9vGbgqN=m%EGsT6o@Ya9?UP@a?*oNTI*~vI5Y@R8YV?wghPFd1X?gK~o_AM+fN81) zp?42wqd}|pIiI(`1U}%vzL~OepDzR$WZ19wpmnzYr`6JF}!4$l; zx+x@=JXO-EtoSN2$><{$;x%#%8!etI(;WA6PN7Q@wXvz#MI|g^pLy$gF)jxXn@9*b8%7v5|lv$4*wMIZ6@p4;fiV?S~h@AN1Bt429pirtC94|Kg;l2X`>!3PexkgQ2 zNH1~lAol&(#FfgGpryq*z7X2daH zfU!0$)b{!A>e-IMhKqjv`nBqEjG+hjyIIyh9k~gbZT-gsK;3XA5fNCU>RR3`U?t>{ z0+Zt#8S$wMKNiv@HlVy*Yl*}Pmw$^y80nuvkwQ89qVeE65G9UB2p6NTc{t}dT>gVz zQjmbAh)klj7Xl8u3m>n^A1_`q$RvQlFxG$@1r{^~jF$h>z8wQX>_1?Ll<~2p@@>s} z*dc^P#USzytFHY4*bHGfN!usVp9n|T7qWq$MtfTZP(ofqEpdg`hu&=qjB~0sK>eUpFilRl=dpXn=>rBQ!TPSv)0GXG z@6Gt*!MOag`Xf@f9=L$gKpjJ3#Fq-oLO7(U@*vRZ9E)#JpD^kUpxfIFiG2!U)YZ|k ziZV!xl-jp+9d}(%W?ZcNEh6CLl7jg4&=2TtXlO+`>5r#5LcxDA=?d~+mi>G z2gP^5M*BaT$NY$hhylH)OFh*6^zkuZ1LhTgf!eKUu_=Vi+6&-bGU(p-{oa;feeFA2 z&rVB3k3^B%iRfQ;_cJyR3ExjmhBtcg%fp`sFeLbAM)}|#Gi`@fYNc688xdFe=Opv~ zW2X^i;*_WHHkTiO_apd62RUVr$v7dP{{xr1atJe~whsDykXG+&7jK4!Ww(;h?r>Pg z!$QX;GQzN+`iz)i8!ljy9-JP2?S~arBd&#Lkd{wc)IK012L^r6L#~R zGV7kAlaq?1nwb9b)_;#VtRF&K$!9DJA*Tb+`Rw&+4fhl$%_vHC2N&og-K2JAoU$`-JgrLtr7q7dQLxBj3`C8J7GvwC^;1DiOo^oui2}LQ)E=2w!JSEToaXZk;av5+^J4p zw|_Etjr{0q-*mpk^}~V5_kk1flEkrnNlH@@69SUVqdDT5Q;nMxL7_B0iM&vwHT16W zME3TM#-Pr13O`0a&+mRjO3Mh$I_iV4FIXU5*{lPRhd&}s)$52IB_+d+bz!Iy%O@Zw5>Y;h5|wU!uGBPwNae4cO@iJC1IN~s z@_Aed>?|vEmIcwd+st>!Q$bxn4+O;8N1!pp{M_<<62({x$avxWAJ`_0>7;y^phRBc z$xQz718*ad(2qr13Lc6-fa-$#2CEVwh=(oTQi5FC2LmmH3(&k$ROf+ySyKSPa9FIU zz4VLu2(`tNLhDdUAEEm1G2}#TDq&@#+aELbIXzftpL@6-f4zUZK-QCHuzk^$VO3^R z+5v^(MC_y%;vwJk{hK*>(tW|7BM}zUs3e=ySGTZqg}P(}9%t4l5X*6)t?ZHVxsOH< zX$wXk*_fraYiyuP{TkuPSm8Je^fM`phDH1Es+kAixQfs^8cBn+r$ux7)?;J+(v+Z~ z{*nEc=(Bi@nW8>8b{(s2k5tiZGrf1ec4kHTV*fi?cqbp8Pu8`zvGh12#I*0bLYR_C z=pBe(fW#H95H?nY|3CP~$$)@iAo%V(?@u*V)iHiRlMfEdlm+3UB(&QqAu`YMwVy_o zyXp5{3BGMx&_seIvgOBipOsfsjI)gVV}h>SiOP9O z^$bEJ$@r|f4f$Det5z4~k?!9b`3mfNA^H7wn$=-=wo9ri_qv#nG1VkEj}?WX%^f$@ zJznoa+u0%TN7^|ciG3$vU3CYQXf+y%cJiRkf?4cC17|Cf@(v|xn@xUT&k`khA~ayR zB7*t&DzSqqBat^~kky%K;}a5YO}cn66FMr{Qs*tV?KZx zQ41#LKa973hANDo|BBCn;i#G>ZT72xud|J_KIWCct*A2#+Bn%ewCUI#CN) zLmd%jNG!}>wYmV{Die)5^DmWBEw;bvUSPk1A9RXii?e_!)en1VMW=j?sr>#Fa1r|Z z5qoT>xsZk%!x8GsX;vQ*4PF`fb%MQ1pyR8kH&X0wvFaJUg&V%86rKGMdt4+pH~^sW zsq+X#w#r4g=7QAK)&0YNv^(`|2KaatDQN1jb$TLQ&8{bb9{pBVUlBufVl7+(Lry|_ ziKM~n@vo^}BCg+&#fYtGPty7s7LQzY1o=y6^8;EbdfXHn6{%ZNg|1Sl%?2e@|FX;j_3 zJa!Gn9ZNIh=(8zEbKe*D1e%TjaIes{YOZtGRxoAoW6k|i#e(t@1&u9{gbn0Y`L1P zi*KdXLj5X(Zf(~~aBrU>91H(YNyS;n-Vm6&-eqU!PO0>IRgcVp)CpT?LhC zSCPGQmX&`7sapX+W#FfO)->*^uxc?&a&<6obdUc3+I#YND7&Z`gOP14nJh8($SxHz zMs``FEKT+$N{uahjEEw;EMX#9%1%Z0Eo4i`2pMT&(#sT;tlu;B*4r}QKi^;9ANk#% z-!soVcka38p5>l%&pA&Rklx88V=-@t9YAn(Mwl`ueIoA$?VqgaB0jLKTn#EUy^g>R zcYVmFzGPT7Rx_TCc5uk+ean|lxa|Fs)W&+rwA{AIBfrTnaOFb=I5~<6Rn+Q7Q~IT9 zEngA8yeDiNdba9t_EtIB>`7~ERCS7q?c&UtJ!ZF_jBmK2GJ8qznLiz(b7m#?GdNbI zLZYolc?s_l@8`7w3wsZHqv8y!inAc_z9i8(QC~ls?epH~b@Z)s6&x1@)hb(=s0#vN z{T&gvX0|^)?8v${t|VXTp!1b-O%q)OTC zJNqRfeLRmCfBE;VnAk}-**e73v4auKK{V<>MZbAi38qm zaJ@@91m^vG{9F0C>p;U9W+ zefUc-UeZNnXSy;siWgGn-8!QXInWgRna5IL?R?v38Lk_VEp{PAo~?bNArIAoJual2 zdyW4#KT6o%oyn4|n1gzT8!|v@xV|iFYmjG*q9JX007dT{4>T)|)caO?4y|%T$d%Jp zzG#_?KLQ@d5J6n+8?VLH-OFR4EweK7>x&hKo)6EgKY!TY$KRiY%wo3t^2D-ViFNKW zB<`ZU(e;4Z=*411sZOO~YxU1xt@gc;0Or>V3PO7aJ-(l{aW00|X{c;Mg2jysaJO*e zltMsih`__O)9mHx@7Yu|Kzv#A0e{2FUg1wjd@W;v!}>>9Gl@2ku-{ICBycUjkUp9g zE=7QgXh^jkaIdqkd^OyS_$05`wXP|!p4jUBlx_{S1$d3lzo_z3{L;DVy*R}AMUYC< zw4Dyyb|H9j5**tYNQ2S_%8Dh}tsT%t&dhun^yjhbC3~C*fUPP&7qsMqJ^M`X0}}Z- zHSio@ji*drXqmQB?MNVfZvL!RGpjAbm3H}rpp>M8U+B)(=f2>X4O?^89YKs1TP4?G z(u;@ZdGL<3M`gBM-+OmF(hU!DuzDE=crGBpf#CfijZ~NfG^FDop3q13Y(X{&LXAM; z^&weQ^ZHoD6IjcdHZTMy>Jbuo1$S*(1nd}+;JDiqR(>MmALzj8NKX#OWgu3DKsHK< zu=OivnKo&i_yQPSGZ6>`@8zb?JT0%t#^3d@i)&)h5Ec!d{OUuU>sbqB*nj<{fPkqI zO}sObvCyORz|i^>805KCM3@oqsfgH2mwhQZVI3mNM$LU&AMOf9kUlnf;|(89HQm-< z0xBoDeykQ^*HZ#jrbd0_-^8?MJ(m$vnm&xrq8=%_clU0Dd0)<#*-02e(+s9{s-gI% z;?U7$P55Obcu@_aeyPU0F*KZ3VSU#t+DmLl{Ea@yU*$QKaHpMT90f6J0SJAo@?tn8TqP858tlO>sjNav(( z2%8BWamk08^pjn?aH}kAOefyK(c3BLXyX~ue`)z;q z^MB&8*4xqO6ZNZVV-gAqyV@nZ^PlD2Y#y-;WHXF5@B3O14fyF_o0bTkE7F}0#>UC< zihR}et8S=?C!k-OOD+(>YQt*$dawVrAPzOuf*h0mZ!+4|#bD^tI z>znoWGi2oJL|gEth{L0&h7~^9Hn$lp7mnZCdFN@s25F&kBwPf7hW;D|h=Mx>n)QZQ zS{O{GN(5?v0QjHDtEJ23p8tqG*0U2Y@vzIq<{^D=x}Tw zWH#SBbA>)JQh30q9Z_}6-Jo^c1uPrx9`r!D)L9h0a4sQawIw+De(*ck@8m(AL+gL# zSi)y`Q+8e0~`-x7vWVQK6bJPmwk8}bjkM&;# zd@RZiPI>jvo@CM?$JIuA)(1HS&jm)4lWjsy0ZFwKXE_dRYG{6x+mbgZ|@8Gb27l4 zF$7GhRd)w+a2CbJO_pACIqy1I$K0-UisZZz*>0?)r}!wN4!lp7H4iftF$=sPhkQ&n z7^4gz(pVjrrt@@c6PN$`GD9@!tKinJbk^9h>s6;#KtDGXC z5RZPy)(9@_DRDYPkqq7-;V4E8(NWDe;mO*-RKA7mWU>Jjk9T^Fp1E}Px5`swZlni0 z1S?lI2Si9_i7ToF{d02unUf)(_Hhoh9j$mEwc*>Eu0gWR|3~Xj4?o4I;aQt-Q`YMhyC(lCL7X#coa`vlj~@Inc*+JMS%g8wvDG}X2AEHEKdN8V4rH6o z2Gsm&)&j!GDD;G9GB`&qw)IY({v=HUF@+9>r(_#yu0D#x-_7dvy@-QRWD6WX*pc;x z$KT14Wv6#az91=sV4|Zkr=hSsI&#C8>?UC70^v}1~+oG zuV$t=i^oh;>=PqFUBSg(oSlp|6s)QKXBHXrU7LkqiLcVZA4*#G7VaFSqu5HOTw6X<bloU&m+o7Ed`DC*sP4{++;k)-XM_co9fP{#IrwlGith_Zg?2Tt`;=26aCCe@u= z;@#Q=g;%vci4?dtv5KrTN}`?LVKt94O_dFbYNz9_u8QIPxHT%2iJ{HPDwKx++zw>D zoH?jNw;ClQtbiBsiJ|cONRS}N`utoS62$!aXm}m$0sz>Yk!r`}ig5~_^D5KMWju)j zhT#JY0r#Oy=|Or~*i{h$e%e_Pbq)7%wV_j0y}eg!d4=e|9xVSk;0XII45>CCqplLD zY9K3R_k2kZAG-hXAxC~Bxm?np*wRx3`En<@iXR|EYib-Om^gSa!;dwFd54JJuyIRl z5~b89gPHnCTTNb9<9j&%#QdvavB@5L&`WKCm^i?CBrHwz*Tdl7fFqi;&LI@#?B5C! z!(OHMM*<7AW zYk=Dy@aY`^z5JNkPgN3KFdb-ohuREke5WRjF7%@=NL*C=j{q5*@E2#v$^MUZc|t(Oq?h_S1PA! z&tcMQ^4>UbP}_A?q%U-Lc0m-{HQPTvnF4qUHvm~mP4`xid!sv%lcors%~>SqqoTn$ z+pj#%CqW6gaJ@P{LrIZDD^UGa&)hOM9*k5%6vjozJqE)TU-ek%DWDUWqHn~ENYq~ zJYB^r#sBh__~VDB#dQN^6PrHk-7j{z_79m|ce1?aB`^`BiOpMmYP)YN^|Ogwrt6ft zZd&=wlDY2A$FyWG}#}w7Y|r?5_$QI1cF$; zZyaOdxwKmOrpTbqTN{Di&Fbvqa`6s5iz+c_A>KL2_}lgyd5xEXkdZ`_?*MvKk!bt|11FF2zXXMwOd zduFKOl3UKQhldPTv67Hzcoi7F>9^CZ66re?$0NOvZ1LGDhbXkZq9)d#>B} z*q7Z(4rDRU#U#D|lQ1;Yrov9*>+!{}!~Bpkoe@A#M`wi4Vun0*MPH1P-^1SgY9$0g zv)NZnvmrjDIjL<`KNE3!e1++?pR><9kF1S-R$gfFRQ?x?VssZbkDPx7auK?beRrSg z^ln9?E%UwTTBQ?r7CTPdJklkiZOrP-V&dOjmoUYQ-WVA&fpF~zVXth;mWjsw<4h!kdJ02gZ1x%rhytzM>dg)Shf{TkGZ zz8q8N+FFEkUHLTi@rAH|5NTgXHy0_yctkx#(JHASTZjI*PyxM913-{Hocwd~Sx!3< zO1c&5xH0=oqEmLsgPpdM5l0SMT16V6nH0Tqk#9*4(n1k*j77-ohvF~8*{N3?U zPp6}SM}rRzqz!X(PbJ_w>|NS&?798RwuyQDd8>F^boXiVV#71Cjyn<7s5DxW^FTkC z)Tno9BDSI0_TE&KY8Nw1%SB9&wUm@D zu`dZ*63nIX!rn472A9YB2^XGDKVSVgnOolnzKMM>-{bN``wRcX!EASnMswzj zj-LAbJ6gv6&o4|e4ejN3TA@SG5FP3x*xL5?p4e#vO26UkVv@4C_f5fL4wJLcbaCgdQBZR~^fEgT8HpAm)&SY-f9wl>+yRm;cC~v& z4ix_AM&SaU1ni&w0`(VD;J-8l_@w`GAK$A1MGgA-B4F2^c9T%gU+#T>DiU-B9V4y%dl&N8n4)mO z&CzkOKYkDR^6zjzBls^e|B}jIzWa$I{?AfHZYY}6HU2`#77%b$`r5`? B@LK=? literal 0 HcmV?d00001 diff --git a/docs/resources/nvidia_logo.png b/docs/resources/nvidia_logo.png new file mode 100644 index 0000000000000000000000000000000000000000..578592cfc322f51bc2e717e50a6892d063011732 GIT binary patch literal 4255 zcmZ`+c|4SB`yU2lFH70C8H^KV7-bz}FWL94Br%xmj6wFKl9*#eWM57QS&}tdM`RaS zqA^~CFqV@cTi#LUJ*VIKoZtO??&rFm>-v7L?{z=-{m1jfo0(ir<% z)|fgGj11H}UxZB&)zG&8+U~NOu`%Er)n^3I(QpEQRF8&w0ciLEC)KDP07WD4mw)jd z20-^`+=OaRhCX$k;C@(O9?joX@@W4;8|MN4@)0LAWtYPys$ufew+{pWSh!9m4d6)@ zF91OM2xDc3wKFzSb@BCvIlKDaK*K`3{Z3c_WQZ!&^+sczMMJ#3d;(QN)WP2os#O0Z z3{F6?7QU`lrv3{y>cyMqq zELZ{N8{iI?S5Z-c%OT(hge(;y8yMmzzn*ZGET z5LO)wJ`wtB{OA*lar;}zC-Bc@Q5Oh5$-w1da`1nnp+hkL5A7uLgZ6!0Kje@n#Z>Jr z(1E^QK__jYln}`83jZhgcYi+!cK=5BIrvY4X#fUIeI?%&bx?Q^{<(KGIe~td;=`~oL$hU|51Jl{S*CN$Np~}1$pJ4b$-hH$wb0W z_U)&A{BcOWN2#X-#f*gib>dLW0?O$x007pl%eq=t9`s{RLLZ{_&UCr+-Zkc6d|M?Y z$HAKU7OaYpiT$1PPTm~t-E=9YcOG+RQ#*By??`QloZ&wG*!i7Q0hm!r^fjHXZh;o? z^;@tuevY=ltH$WyV61Gzq-C0@v+JdvwGd_+KIYuJ8uwtfks znfz*?-R8PLm4uv0X58LW@1S%ed^rO1tFvN5y1Q7J1jZB#iUqcpwHtQSU6;qPNOdz- z*J-MN)|lDp?nOs*c;$%$vv+IJB!d*29*0V|)JqtKrU{EzY=Ue!PzARD_owt5%^Z~F z+A9}jUYLGlie!DbXYfENhAWy*a{3_`qV)OiVEepG8H&m*WY%W5cl3M?U7v#lCDW!B zxu4j^F*$k5)9~Q*rqHkHL*`Ko8S^N0`EuW z{Hy3bS{t@8?l4THWAL{@F^csu(|m<}nq3zl>S@Bqf0W-DoIk*9n&c@%75whIs@W7O zL+9byRz{K^+{Slmhv>-c8YXILT0z@*%}ga)_^ye^5J;M!@hkr7IQyBL0PHIu9VM=G`7RY&DNgd%g`8oH z?xdT^YG1?QQqbEq{l#hIi#Jn3kMOrN*q>Ral?UD!;VsC1?_ay&dv^d_sQS3~W17sA zTil0gt(DYT`dTPw)Ud#GoI7C(Mibe=oquid;%DAZ`WA!qZ43|m9gYjGzQK1281l!o z-0$Pa`^bJsr`ylK!`(=DuxohMGU-&4w%xY(Wm0BG_N;>++hzz6IMs08aC%F3f%zO< zLHIWT0_)nk8MZi9KP*nSzUI@{B94j{_>A5(q)55J#6Fs3nsHMP-5Ttu5!`qh#oi=Bt2dH`*MF73KdqhRz2CRt&`WHGX!h_kRFp>DP2d2% zZSm9QDsFK?OzsWP1up&eQIO=T_>Fy7<3@d9Gkq2a2Oyrptq|WmOHd4)4qMTwg^2aT zSwD@qUHeor&imoLV^wH_9Sqv6SZpN&tze%m)GcZOT4+>e*dZXn@uc#MM==t}NW;=x z+C#U#YLz=p0jHEfYZgm#x>v_i*UBi6pl1a7I}JN@*lwX|*N?*Q2SOxTW5*XX=4}yV z>9L`>r#jZSZFe?jH%+zD(evda!tU1YvBvzQk81a<59&gKCEQs3t)1p(tMXJst>QT6 z2A|atwuNEHg0<_UTb~y|y#W&IfFDAmf*(9Z)?b?AP!Oy&vQTz4Ym})|OhF5dBiS`JR_QSi?66xwb z@tx8gA^%C&thhHWd)3bed}z9+IKt(G9>rIlyR-KV2B+Z+!|BYS`5_UnYnXQ8kgG#b zouasC@6BGW5LVVFTI?(w%QaUG7R@43Wd5nTQ$&{4k7vz}6SJ+e!n*J<^tKR2 zxbav2z?Z#q-cQ#v9X9x#b~|S#Ms{iRnq>y-q7;T_d`DcDz9c_&lJqFYZHZjSx!rmk zwi|-|FgxDOo6)whv}kio$aonE-fz|{YGbElj7Hb4r0bC%YFS*kEA|=}dPes%M8C!z zzQPePzi}mz=y{0DGfAt({U?}fDY`$UiO;5{Y^7CKjdRYcK1f6@peXQ>qmCk}b4z%@ z*ZP?b(`i^*iVeuLDrH-PAODLH{qjxORk2BndKH2KKSnM6|EfB~u<)omQs_1ORmSO`G4;!p7 z*{iF`h`(y*eGuGRJqE3lr6Fm4Hb<3IXO!H$o5O_j6O3zW?cv5RVnhj{AJSeu41-f9 ztfH124h`F9KoOGB6DD1o8v+K37X%P6jD+K=W&$Wu$X+oeBDX?O;?29q#DZ}3uTV2V z{Sk#cjVEpN$aA|iJ?6WWNV_SiH$oh-Uk;0meG0Q8zRu1qAG(^61X7l3L4DqX?=`^58p6~IJMsrIK?EWe_$}&OSusFA^mHT?cBm7G?t`D(_ilOgo1N+D$1_s z&xy`Y^ytp3G#%Gk-p7lSsP-sxrP%PKL6$y+TnQ4vHzX`4c#9?BE-_Qo+kct8xLJt% z!#f+LfYsQHoc+DHTM=yjdJVxvKwDjc(}zi%r63;HS{Cfi@Pku!T19>Dl_tekm}CfI z*zfuqcEKMU%oYszKvOrKRwB=K;_HUo9T%;SIF{3HueN_`&sDPpw$UH6@81Y_w)yn! z!GLM$cDG*Ei6J+?RoxIyidN`(+JzfIrQ#`Qx(L>c&j-*Ndp1 z0ME&B?`_51e)T`PEmx*s|4dJKF_1%w^x(qNB5a0Z&`h;k#KD_I$<{NjUrKQqo)LP4 z(1*NH7TUEw7Vq&u71f?-K_L*XgD@eK!Fn53v{$T5)=tuu2rFgpY4ZrBGo{N^s#dKT zh8EsU%B|bpL+;XV0`7~ym~zQ!-Re!M_2EGzpGnmnX+7j(4H97)r0cd7A^w8DtkvCvCrWtxQE3aPBeo60<3YW zl!p`Id`bdE0jEm!j9f$BORW5)*0DHtKIvJA5IWS}j4i8DmyH%Bs(~_*}}hePi1pln39V%~<1D zJE$k8s^#b?JZCFNBf3uz;{&41LR|5HgP3(#(etZ#a{kv0dN%{m_8ue=r*pgI7+B9T z3FMvdHD2BO{qz{v?a>GIv3aGnHj--ur*%syqPzfEbM*LI%1~I$ttJ|p4-RIiZZ~qr`p@ixH literal 0 HcmV?d00001 diff --git a/docs/user_guide/nvflare_cli/job_cli.rst b/docs/user_guide/nvflare_cli/job_cli.rst index 78852af370..3fde8bb39e 100644 --- a/docs/user_guide/nvflare_cli/job_cli.rst +++ b/docs/user_guide/nvflare_cli/job_cli.rst @@ -208,3 +208,40 @@ and change the app_1 batch_size to 4, app_2 batch_size to 6 for sag_pt_deploy_ma The app names must be defined in the job template being used: in this case ``app_1``, ``app_2``, and ``app_server``, are in ``sag_pt_deploy_map``. + +*************************** +FLARE Job Template Registry +*************************** + +Below is a table of all available :github_nvflare_link:`Job Templates `. + +.. csv-table:: + :header: Example,Execution API Type,Controller Type,Description + :widths: 18, 18, 10, 30 + + cyclic_cc_pt,client,client_api,client-controlled cyclic workflow with PyTorch ClientAPI trainer + cyclic_pt,server,client_api,server-controlled cyclic workflow with PyTorch ClientAPI trainer + psi_csv,server,Executor,private-set intersection for csv data + sag_cross_np,server,client_executor,scatter & gather and cross-site validation using numpy + sag_cse_pt,server,client_api,scatter & gather workflow and cross-site evaluation with PyTorch + sag_gnn,server,client_api,scatter & gather workflow for gnn learning + sag_nemo,server,client_api,Scatter and Gather Workflow for NeMo + sag_np,server,client_api,scatter & gather workflow using numpy + sag_np_cell_pipe,server,client_api,scatter & gather workflow using numpy + sag_np_metrics,server,client_api,scatter & gather workflow using numpy + sag_pt,server,client_api,scatter & gather workflow using pytorch + sag_pt_deploy_map,server,client_api,SAG workflow using pytorch with deploy_map & site-specific configs + sag_pt_executor,server,Executor,scatter & gather workflow and cross-site evaluation with PyTorch + sag_pt_he,server,client_api,scatter & gather workflow using pytorch and homomorphic encryption + sag_pt_mlflow,server,client_api,scatter & gather workflow using pytorch with MLflow tracking + sag_pt_model_learner,server,ModelLearner,scatter & gather workflow and cross-site evaluation with PyTorch + sag_tf,server,client_api,scatter & gather workflow using TensorFlow + sklearn_kmeans,server,client_api,scikit-learn KMeans model + sklearn_linear,server,client_api,scikit-learn linear model + sklearn_svm,server,client_api,scikit-learn SVM model + stats_df,server,stats_executor,FedStats: tabular data with pandas + stats_image,server,stats_executor,FedStats: image intensity histogram + swarm_cse_pt,client,client_api,Swarm Learning with Cross-Site Evaluation with PyTorch + swarm_cse_pt_model_learner,client,ModelLearner,Swarm Learning with Cross-Site Evaluation with PyTorch ModelLearner + vertical_xgb,server,Executor,vertical federated xgboost + xgboost_tree,server,client_api,xgboost horizontal tree-based collaboration model diff --git a/examples/README.md b/examples/README.md index 38858563a5..04da856400 100644 --- a/examples/README.md +++ b/examples/README.md @@ -77,7 +77,7 @@ When you open a notebook, select the kernel `nvflare_example` using the dropdown |----------------------------------------------------------------------------------------------------------------------------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Notebook for Hello Examples](./hello-world/hello_world.ipynb) | - | Notebook for examples below. | | [Hello Scatter and Gather](./hello-world/hello-numpy-sag/README.md) | Numpy | Example using [ScatterAndGather](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.scatter_and_gather.html) controller workflow. | -| [Hello Cross-Site Validation](./hello-world/hello-numpy-cross-val/README.md) | Numpy | Example using [CrossSiteModelEval](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.cross_site_model_eval.html) controller workflow. | +| [Hello Cross-Site Validation](./hello-world/hello-numpy-cross-val/README.md) | Numpy | Example using [CrossSiteModelEval](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.cross_site_model_eval.html) controller workflow, and example using previous results without training workflow. | | [Hello Cyclic Weight Transfer](./hello-world/hello-cyclic/README.md) | PyTorch | Example using [CyclicController](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.cyclic_ctl.html) controller workflow to implement [Cyclic Weight Transfer](https://pubmed.ncbi.nlm.nih.gov/29617797/). | | [Hello PyTorch](./hello-world/hello-pt/README.md) | PyTorch | Example using an image classifier using [FedAvg](https://arxiv.org/abs/1602.05629) and [PyTorch](https://pytorch.org/) as the deep learning training framework. | | [Hello TensorFlow](./hello-world/hello-tf2/README.md) | TensorFlow2 | Example of using an image classifier using [FedAvg](https://arxiv.org/abs/1602.05629) and [TensorFlow](https://tensorflow.org/) as the deep learning training framework. | diff --git a/examples/hello-world/step-by-step/README.md b/examples/hello-world/step-by-step/README.md index 77ecc34f52..aba2957283 100644 --- a/examples/hello-world/step-by-step/README.md +++ b/examples/hello-world/step-by-step/README.md @@ -1,24 +1,22 @@ # Step-by-Step Examples +To run the notebooks in each example, please make sure you first set up a virtual environment and install "./requirements.txt" and JupyterLab following the [example root readme](../README.md). + +* [cifar10](cifar10) - Multi-class classification with image data using CIFAR10 dataset +* [higgs](higgs) - Binary classification with tabular data using HIGGS dataset + These step-by-step example series are aimed to help users quickly get started and learn about FLARE. For consistency, each example in the series uses the same dataset- CIFAR10 for image data and the HIGGS dataset for tabular data. -The examples will build upon previous ones to showcase different features, workflows, or APIs, allowing users to gain a comprehensive understanding of FLARE functionalities. +The examples will build upon previous ones to showcase different features, workflows, or APIs, allowing users to gain a comprehensive understanding of FLARE functionalities. See the README in each directory for more details about each series. + +## Common Questions -Given a machine learning problem, here are some common questions we aim to cover when formulating a federated learning problem: +Here are some common questions we aim to cover in these examples series when formulating a federated learning problem: * What does the data look like? * How do we compare global statistics with the site's local data statistics? -* How to formulate the federated algorithms - * https://developer.download.nvidia.com/healthcare/clara/docs/federated_traditional_machine_learning_algorithms.pdf -* Given the formulation, how to convert the existing machine learning or deep learning code to Federated learning code. - * [ML to FL examples](https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/ml-to-fl/README.md) -* For different types of federated learning workflows: Scatter and Gather, Cyclic Weight Transfer, Swarming learning, -Vertical learning, ... what do we need to change ? -* How can we capture the experiment log, so all sites' metrics and global metrics can be viewed in experiment tracking tools such as Weights & Biases, MLfLow, or Tensorboard - -In these "step-by-step" examples, we will dive into these questions in two series of examples (See the README in each directory for more details about each series): - -* [cifar10](cifar10) - Multi-class classification with image data using CIFAR10 dataset -* [higgs](higgs) - Binary classification with tabular data using HIGGS dataset - - +* How to formulate the [federated algorithms](https://developer.download.nvidia.com/healthcare/clara/docs/federated_traditional_machine_learning_algorithms.pdf)? +* How do we convert the existing machine learning or deep learning code to federated learning code? [ML to FL examples](https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/ml-to-fl/README.md) +* How do we use different types of federated learning workflows (e.g. Scatter and Gather, Cyclic Weight Transfer, Swarming learning, +Vertical learning) and what do we need to change? +* How can we capture the experiment log, so all sites' metrics and global metrics can be viewed in experiment tracking tools such as Weights & Biases MLfLow, or Tensorboard diff --git a/job_templates/readme.md b/job_templates/readme.md index ba4e6fc61f..6963c6165f 100644 --- a/job_templates/readme.md +++ b/job_templates/readme.md @@ -13,8 +13,43 @@ Each job template contains the following informations * information card: info.md for display purpose * information config: used by program -# Configuration format +## Configuration format Configurations are written in HOCON (human optimized object Notation). As a variant of JSON, .conf can also use json format. The pyhocon format allows for comments, and you can remove many of the double quotes as well as replace ":" with "=" to make the configurations look cleaner. You can find details in [pyhoconb: HOCON Parser for python](https://github.com/chimpler/pyhocon). + +## List of Job Templates + +View all the available job templates with the following command: + +```nvflare job list_templates``` + +| Example | Controller-Type | Execution API Type | Description | +|---------|-----------------|-----------------|-------------| +| [cyclic_cc_pt](./cyclic_cc_pt) | client | client_api | client-controlled cyclic workflow with PyTorch ClientAPI trainer | +| [cyclic_pt](./cyclic_pt) | server | client_api | server-controlled cyclic workflow with PyTorch ClientAPI trainer | +| [psi_csv](./psi_csv) | server | Executor | private-set intersection for csv data | +| [sag_cross_np](./sag_cross_np) | server | client executor | scatter & gather and cross-site validation using numpy | +| [sag_cse_pt](./sag_cse_pt) | server | client_api | scatter & gather workflow and cross-site evaluation with PyTorch | +| [sag_gnn](./sag_gnn) | server | client_api | scatter & gather workflow for gnn learning | +| [sag_nemo](./sag_nemo) | server | client_api | Scatter and Gather Workflow for NeMo | +| [sag_np](./sag_np) | server | client_api | scatter & gather workflow using numpy | +| [sag_np_cell_pipe](./sag_np_cell_pipe) | server | client_api | scatter & gather workflow using numpy | +| [sag_np_metrics](./sag_np_metrics) | server | client_api | scatter & gather workflow using numpy | +| [sag_pt](./sag_pt) | server | client_api | scatter & gather workflow using pytorch | +| [sag_pt_deploy_map](./sag_pt_deploy_map) | server | client_api | SAG workflow with pytorch, deploy_map, site-specific configs | +| [sag_pt_executor](./sag_pt_executor) | server | Executor | scatter & gather workflow and cross-site evaluation with PyTorch | +| [sag_pt_he](./sag_pt_he) | server | client_api | scatter & gather workflow using pytorch and homomorphic encryption | +| [sag_pt_mlflow](./sag_pt_mlflow) | server | client_api | scatter & gather workflow using pytorch with MLflow tracking | +| [sag_pt_model_learner](./sag_pt_model_learner) | server | ModelLearner | scatter & gather workflow and cross-site evaluation with PyTorch | +| [sag_tf](./sag_tf) | server | client_api | scatter & gather workflow using TensorFlow | +| [sklearn_kmeans](./sklearn_kmeans) | server | client_api | scikit-learn KMeans model | +| [sklearn_linear](./sklearn_linear) | server | client_api | scikit-learn linear model | +| [sklearn_svm](./sklearn_svm) | server | client_api | scikit-learn SVM model | +| [stats_df](./stats_df) | server | stats executor | FedStats: tabular data with pandas | +| [stats_image](./stats_image) | server | stats executor | FedStats: image intensity histogram | +| [swarm_cse_pt](./swarm_cse_pt) | client | client_api | Swarm Learning with Cross-Site Evaluation with PyTorch | +| [swarm_cse_pt_model_learner](./swarm_cse_pt_model_learner) | client | ModelLearner | Swarm Learning with Cross-Site Evaluation with PyTorch ModelLearner | +| [vertical_xgb](./vertical_xgb) | server | Executor | vertical federated xgboost | +| [xgboost_tree](./xgboost_tree) | server | client_api | xgboost horizontal tree-based collaboration model | From f5d2a2f36de940d532edc85a8d7636d579f19f1a Mon Sep 17 00:00:00 2001 From: Sean Yang Date: Tue, 23 Jan 2024 13:30:47 -0800 Subject: [PATCH 13/39] Rename to execution api type, update job command output (#2306) * rename client category to execution api type * fix output format --- docs/user_guide/nvflare_cli/job_cli.rst | 95 ++++++++----------- examples/README.md | 2 +- job_templates/cyclic_cc_pt/info.conf | 2 +- job_templates/cyclic_pt/info.conf | 2 +- job_templates/psi_csv/info.conf | 2 +- job_templates/sag_cross_np/info.conf | 2 +- job_templates/sag_cse_pt/info.conf | 2 +- job_templates/sag_gnn/info.conf | 2 +- job_templates/sag_nemo/info.conf | 2 +- job_templates/sag_np/info.conf | 2 +- job_templates/sag_np_cell_pipe/info.conf | 2 +- job_templates/sag_np_metrics/info.conf | 2 +- job_templates/sag_pt/info.conf | 2 +- job_templates/sag_pt_deploy_map/info.conf | 2 +- job_templates/sag_pt_executor/info.conf | 2 +- job_templates/sag_pt_he/info.conf | 2 +- job_templates/sag_pt_mlflow/info.conf | 2 +- job_templates/sag_pt_model_learner/info.conf | 2 +- job_templates/sag_tf/info.conf | 2 +- job_templates/sklearn_kmeans/info.conf | 2 +- job_templates/sklearn_linear/info.conf | 2 +- job_templates/sklearn_svm/info.conf | 2 +- job_templates/stats_df/info.conf | 2 +- job_templates/stats_image/info.conf | 2 +- job_templates/swarm_cse_pt/info.conf | 2 +- .../swarm_cse_pt_model_learner/info.conf | 2 +- job_templates/vertical_xgb/info.conf | 2 +- job_templates/xgboost_tree/info.conf | 2 +- nvflare/tool/job/job_cli.py | 18 ++-- nvflare/tool/job/job_client_const.py | 6 +- 30 files changed, 78 insertions(+), 95 deletions(-) diff --git a/docs/user_guide/nvflare_cli/job_cli.rst b/docs/user_guide/nvflare_cli/job_cli.rst index 3fde8bb39e..94f0395198 100644 --- a/docs/user_guide/nvflare_cli/job_cli.rst +++ b/docs/user_guide/nvflare_cli/job_cli.rst @@ -45,22 +45,42 @@ the job_templates. The output should be similar to the following: -.. code-block::shell +.. code-block:: none The following job templates are available: - ------------------------------------------------------------------------------------------------------------------------ - name Description Controller Type Client Category - ------------------------------------------------------------------------------------------------------------------------ - sag_cross_np scatter & gather and cross-site validation using numpy server client executor - sag_pt scatter & gather workflow using pytorch server client_api - sag_pt_ddp scatter & gather workflow using pytorch + ddp server client_api - sag_pt_deploy_map SAG workflow with pytorch, deploy_map, site-specific configs server client_api - sag_tf scatter & gather workflow using TensorFlow server client_api - stats_df FedStats: tabular data with pandas server stats executor - stats_image FedStats: image intensity histogram server stats executor - ------------------------------------------------------------------------------------------------------------------------ - + ---------------------------------------------------------------------------------------------------------------------- + name Description Controller Type Execution API Type + ---------------------------------------------------------------------------------------------------------------------- + cyclic_cc_pt client-controlled cyclic workflow with PyTorch ClientAPI tra client client_api + cyclic_pt server-controlled cyclic workflow with PyTorch ClientAPI tra server client_api + psi_csv private-set intersection for csv data server Executor + sag_cross_np scatter & gather and cross-site validation using numpy server client executor + sag_cse_pt scatter & gather workflow and cross-site evaluation with PyT server client_api + sag_gnn scatter & gather workflow for gnn learning server client_api + sag_nemo Scatter and Gather Workflow for NeMo server client_api + sag_np scatter & gather workflow using numpy server client_api + sag_np_cell_pipe scatter & gather workflow using numpy server client_api + sag_np_metrics scatter & gather workflow using numpy server client_api + sag_pt scatter & gather workflow using pytorch server client_api + sag_pt_deploy_map SAG workflow with pytorch, deploy_map, site-specific configs server client_api + sag_pt_executor scatter & gather workflow and cross-site evaluation with PyT server Executor + sag_pt_he scatter & gather workflow using pytorch and homomorphic encr server client_api + sag_pt_mlflow scatter & gather workflow using pytorch with MLflow tracking server client_api + sag_pt_model_learner scatter & gather workflow and cross-site evaluation with PyT server ModelLearner + sag_tf scatter & gather workflow using TensorFlow server client_api + sklearn_kmeans scikit-learn KMeans model server client_api + sklearn_linear scikit-learn linear model server client_api + sklearn_svm scikit-learn SVM model server client_api + stats_df FedStats: tabular data with pandas server stats executor + stats_image FedStats: image intensity histogram server stats executor + swarm_cse_pt Swarm Learning with Cross-Site Evaluation with PyTorch client client_api + swarm_cse_pt_model_l Swarm Learning with Cross-Site Evaluation with PyTorch Model client ModelLearner + vertical_xgb vertical federated xgboost server Executor + xgboost_tree xgboost horizontal tree-based collaboration model server client_api + ---------------------------------------------------------------------------------------------------------------------- + +View all the available templates at the :github_nvflare_link:`FLARE Job Template Registry `. Setting job_template path ------------------------- @@ -90,20 +110,18 @@ The options for usage are as follows: .. code-block:: - usage: nvflare job create [-h] [-j [JOB_FOLDER]] [-w [TEMPLATE]] [-s [SCRIPT]] [-sd [SCRIPT_DIR]] [-f [CONFIG_FILE ...]] [-debug] [-force] + usage: nvflare job create [-h] [-j [JOB_FOLDER]] [-w [TEMPLATE]] [-sd [SCRIPT_DIR]] [-f [CONFIG_FILE [CONFIG_FILE ...]]] [-debug] [-force] - options: + optional arguments: -h, --help show this help message and exit -j [JOB_FOLDER], --job_folder [JOB_FOLDER] job_folder path, default to ./current_job directory -w [TEMPLATE], --template [TEMPLATE] - template name or template folder. You can use list_templates to see available jobs from job templates, pick name such as 'sag_pt' as template name. Alternatively, you can use the path to the job template folder, such as - job_templates/sag_pt - -s [SCRIPT], --script [SCRIPT] - code script such as train.py + template name or template folder. You can use list_templates to see available jobs from job templates, pick name such as 'sag_pt' as template name. Alternatively, you can use the path to the job + template folder, such as job_templates/sag_pt -sd [SCRIPT_DIR], --script_dir [SCRIPT_DIR] script directory contains additional related files. All files or directories under this directory will be copied over to the custom directory. - -f [CONFIG_FILE ...], --config_file [CONFIG_FILE ...] + -f [CONFIG_FILE [CONFIG_FILE ...]], --config_file [CONFIG_FILE [CONFIG_FILE ...]] Training config file with corresponding optional key=value pairs. If key presents in the preceding config file, the value in the config file will be overwritten by the new value -debug, --debug debug is on -force, --force force create is on, if -force, overwrite existing configuration with newly created configurations @@ -208,40 +226,3 @@ and change the app_1 batch_size to 4, app_2 batch_size to 6 for sag_pt_deploy_ma The app names must be defined in the job template being used: in this case ``app_1``, ``app_2``, and ``app_server``, are in ``sag_pt_deploy_map``. - -*************************** -FLARE Job Template Registry -*************************** - -Below is a table of all available :github_nvflare_link:`Job Templates `. - -.. csv-table:: - :header: Example,Execution API Type,Controller Type,Description - :widths: 18, 18, 10, 30 - - cyclic_cc_pt,client,client_api,client-controlled cyclic workflow with PyTorch ClientAPI trainer - cyclic_pt,server,client_api,server-controlled cyclic workflow with PyTorch ClientAPI trainer - psi_csv,server,Executor,private-set intersection for csv data - sag_cross_np,server,client_executor,scatter & gather and cross-site validation using numpy - sag_cse_pt,server,client_api,scatter & gather workflow and cross-site evaluation with PyTorch - sag_gnn,server,client_api,scatter & gather workflow for gnn learning - sag_nemo,server,client_api,Scatter and Gather Workflow for NeMo - sag_np,server,client_api,scatter & gather workflow using numpy - sag_np_cell_pipe,server,client_api,scatter & gather workflow using numpy - sag_np_metrics,server,client_api,scatter & gather workflow using numpy - sag_pt,server,client_api,scatter & gather workflow using pytorch - sag_pt_deploy_map,server,client_api,SAG workflow using pytorch with deploy_map & site-specific configs - sag_pt_executor,server,Executor,scatter & gather workflow and cross-site evaluation with PyTorch - sag_pt_he,server,client_api,scatter & gather workflow using pytorch and homomorphic encryption - sag_pt_mlflow,server,client_api,scatter & gather workflow using pytorch with MLflow tracking - sag_pt_model_learner,server,ModelLearner,scatter & gather workflow and cross-site evaluation with PyTorch - sag_tf,server,client_api,scatter & gather workflow using TensorFlow - sklearn_kmeans,server,client_api,scikit-learn KMeans model - sklearn_linear,server,client_api,scikit-learn linear model - sklearn_svm,server,client_api,scikit-learn SVM model - stats_df,server,stats_executor,FedStats: tabular data with pandas - stats_image,server,stats_executor,FedStats: image intensity histogram - swarm_cse_pt,client,client_api,Swarm Learning with Cross-Site Evaluation with PyTorch - swarm_cse_pt_model_learner,client,ModelLearner,Swarm Learning with Cross-Site Evaluation with PyTorch ModelLearner - vertical_xgb,server,Executor,vertical federated xgboost - xgboost_tree,server,client_api,xgboost horizontal tree-based collaboration model diff --git a/examples/README.md b/examples/README.md index 04da856400..9b9e0f25b9 100644 --- a/examples/README.md +++ b/examples/README.md @@ -83,7 +83,7 @@ When you open a notebook, select the kernel `nvflare_example` using the dropdown | [Hello TensorFlow](./hello-world/hello-tf2/README.md) | TensorFlow2 | Example of using an image classifier using [FedAvg](https://arxiv.org/abs/1602.05629) and [TensorFlow](https://tensorflow.org/) as the deep learning training framework. | ## 2. Step-by-Step Examples -| Example | Dataset | Controller-Type | Client Category | Framework | Summary | +| Example | Dataset | Controller-Type | Execution API Type | Framework | Summary | |---------|---------|-----------------|-----------------|-----------|---------| | [image_stats](./hello-world/step-by-step/cifar10/stats/image_stats.ipynb) | CIFAR10 | server | Executor | Pandas | Example for federated stats image histogram calculation. | | [sag](./hello-world/step-by-step/cifar10/sag/sag.ipynb) | CIFAR10 | server | Client API| PyTorch | Example for FedAvg with [ScatterAndGather](https://nvflare.readthedocs.io/en/main/apidocs/nvflare.app_common.workflows.scatter_and_gather.html) controller workflow using the Client API. | diff --git a/job_templates/cyclic_cc_pt/info.conf b/job_templates/cyclic_cc_pt/info.conf index f6906f1b33..6777ec617b 100644 --- a/job_templates/cyclic_cc_pt/info.conf +++ b/job_templates/cyclic_cc_pt/info.conf @@ -1,5 +1,5 @@ { description = "client-controlled cyclic workflow with PyTorch ClientAPI trainer" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "client" } \ No newline at end of file diff --git a/job_templates/cyclic_pt/info.conf b/job_templates/cyclic_pt/info.conf index ff5f710307..b5d297af4b 100644 --- a/job_templates/cyclic_pt/info.conf +++ b/job_templates/cyclic_pt/info.conf @@ -1,5 +1,5 @@ { description = "server-controlled cyclic workflow with PyTorch ClientAPI trainer" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/psi_csv/info.conf b/job_templates/psi_csv/info.conf index f6b6d49a8c..6894c5ba35 100644 --- a/job_templates/psi_csv/info.conf +++ b/job_templates/psi_csv/info.conf @@ -1,5 +1,5 @@ { description = "private-set intersection for csv data" - client_category = "Executor" + execution_api_type = "Executor" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_cross_np/info.conf b/job_templates/sag_cross_np/info.conf index 838c9bc165..f9be1549e4 100644 --- a/job_templates/sag_cross_np/info.conf +++ b/job_templates/sag_cross_np/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather and cross-site validation using numpy" - client_category = "client executor" + execution_api_type = "client executor" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_cse_pt/info.conf b/job_templates/sag_cse_pt/info.conf index 310b4423fb..258d56d9a1 100644 --- a/job_templates/sag_cse_pt/info.conf +++ b/job_templates/sag_cse_pt/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow and cross-site evaluation with PyTorch" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_gnn/info.conf b/job_templates/sag_gnn/info.conf index 48882d4e4c..e4aaabd7c2 100644 --- a/job_templates/sag_gnn/info.conf +++ b/job_templates/sag_gnn/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow for gnn learning" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_nemo/info.conf b/job_templates/sag_nemo/info.conf index ddd0fec10d..96b046b4b6 100644 --- a/job_templates/sag_nemo/info.conf +++ b/job_templates/sag_nemo/info.conf @@ -1,5 +1,5 @@ { description = "Scatter and Gather Workflow for NeMo" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_np/info.conf b/job_templates/sag_np/info.conf index e00a60749f..365c3cb62b 100644 --- a/job_templates/sag_np/info.conf +++ b/job_templates/sag_np/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow using numpy" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_np_cell_pipe/info.conf b/job_templates/sag_np_cell_pipe/info.conf index e00a60749f..365c3cb62b 100644 --- a/job_templates/sag_np_cell_pipe/info.conf +++ b/job_templates/sag_np_cell_pipe/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow using numpy" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_np_metrics/info.conf b/job_templates/sag_np_metrics/info.conf index e00a60749f..365c3cb62b 100644 --- a/job_templates/sag_np_metrics/info.conf +++ b/job_templates/sag_np_metrics/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow using numpy" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_pt/info.conf b/job_templates/sag_pt/info.conf index 7e1015daf9..31ded18cf9 100644 --- a/job_templates/sag_pt/info.conf +++ b/job_templates/sag_pt/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow using pytorch" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_pt_deploy_map/info.conf b/job_templates/sag_pt_deploy_map/info.conf index 6c387de8f7..df5ffdb4e3 100644 --- a/job_templates/sag_pt_deploy_map/info.conf +++ b/job_templates/sag_pt_deploy_map/info.conf @@ -1,5 +1,5 @@ { description = "SAG workflow with pytorch, deploy_map, site-specific configs" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_pt_executor/info.conf b/job_templates/sag_pt_executor/info.conf index e9a2aac332..914f92e5f9 100644 --- a/job_templates/sag_pt_executor/info.conf +++ b/job_templates/sag_pt_executor/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow and cross-site evaluation with PyTorch Executor" - client_category = "Executor" + execution_api_type = "Executor" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_pt_he/info.conf b/job_templates/sag_pt_he/info.conf index 93c4edb7c9..1e60edc4c5 100644 --- a/job_templates/sag_pt_he/info.conf +++ b/job_templates/sag_pt_he/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow using pytorch and homomorphic encryption" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_pt_mlflow/info.conf b/job_templates/sag_pt_mlflow/info.conf index 4e9272a588..e4a251ede0 100644 --- a/job_templates/sag_pt_mlflow/info.conf +++ b/job_templates/sag_pt_mlflow/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow using pytorch with MLflow tracking" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } diff --git a/job_templates/sag_pt_model_learner/info.conf b/job_templates/sag_pt_model_learner/info.conf index c2b58b68dc..91df8ec626 100644 --- a/job_templates/sag_pt_model_learner/info.conf +++ b/job_templates/sag_pt_model_learner/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow and cross-site evaluation with PyTorch ModelLearner" - client_category = "ModelLearner" + execution_api_type = "ModelLearner" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sag_tf/info.conf b/job_templates/sag_tf/info.conf index 27e4bba015..363950935b 100644 --- a/job_templates/sag_tf/info.conf +++ b/job_templates/sag_tf/info.conf @@ -1,5 +1,5 @@ { description = "scatter & gather workflow using TensorFlow" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sklearn_kmeans/info.conf b/job_templates/sklearn_kmeans/info.conf index c6d3bf68bc..9ca7516388 100644 --- a/job_templates/sklearn_kmeans/info.conf +++ b/job_templates/sklearn_kmeans/info.conf @@ -1,5 +1,5 @@ { description = "scikit-learn KMeans model" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sklearn_linear/info.conf b/job_templates/sklearn_linear/info.conf index 52ef59cb2d..13774438e9 100644 --- a/job_templates/sklearn_linear/info.conf +++ b/job_templates/sklearn_linear/info.conf @@ -1,5 +1,5 @@ { description = "scikit-learn linear model" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/sklearn_svm/info.conf b/job_templates/sklearn_svm/info.conf index d6678a6e4b..8dc7d5c6dc 100644 --- a/job_templates/sklearn_svm/info.conf +++ b/job_templates/sklearn_svm/info.conf @@ -1,5 +1,5 @@ { description = "scikit-learn SVM model" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/stats_df/info.conf b/job_templates/stats_df/info.conf index ba06c1e0b7..f45779ffd8 100644 --- a/job_templates/stats_df/info.conf +++ b/job_templates/stats_df/info.conf @@ -1,5 +1,5 @@ { description = "FedStats: tabular data with pandas" - client_category = "stats executor" + execution_api_type = "stats executor" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/stats_image/info.conf b/job_templates/stats_image/info.conf index 5e4d21eda8..9c1a2e4ade 100644 --- a/job_templates/stats_image/info.conf +++ b/job_templates/stats_image/info.conf @@ -1,5 +1,5 @@ { description = "FedStats: image intensity histogram" - client_category = "stats executor" + execution_api_type = "stats executor" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/swarm_cse_pt/info.conf b/job_templates/swarm_cse_pt/info.conf index 79c97c66f5..ad0bcd2cae 100644 --- a/job_templates/swarm_cse_pt/info.conf +++ b/job_templates/swarm_cse_pt/info.conf @@ -1,5 +1,5 @@ { description = "Swarm Learning with Cross-Site Evaluation with PyTorch" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "client" } diff --git a/job_templates/swarm_cse_pt_model_learner/info.conf b/job_templates/swarm_cse_pt_model_learner/info.conf index 35d79ca5d1..a690f8637c 100644 --- a/job_templates/swarm_cse_pt_model_learner/info.conf +++ b/job_templates/swarm_cse_pt_model_learner/info.conf @@ -1,5 +1,5 @@ { description = "Swarm Learning with Cross-Site Evaluation with PyTorch ModelLearner" - client_category = "ModelLearner" + execution_api_type = "ModelLearner" controller_type = "client" } diff --git a/job_templates/vertical_xgb/info.conf b/job_templates/vertical_xgb/info.conf index d391599686..99643d8607 100644 --- a/job_templates/vertical_xgb/info.conf +++ b/job_templates/vertical_xgb/info.conf @@ -1,5 +1,5 @@ { description = "vertical federated xgboost" - client_category = "Executor" + execution_api_type = "Executor" controller_type = "server" } \ No newline at end of file diff --git a/job_templates/xgboost_tree/info.conf b/job_templates/xgboost_tree/info.conf index c380f6dc39..9b3f3b0952 100644 --- a/job_templates/xgboost_tree/info.conf +++ b/job_templates/xgboost_tree/info.conf @@ -1,5 +1,5 @@ { description = "xgboost horizontal tree-based collaboration model" - client_category = "client_api" + execution_api_type = "client_api" controller_type = "server" } \ No newline at end of file diff --git a/nvflare/tool/job/job_cli.py b/nvflare/tool/job/job_cli.py index 9b3522c83e..ceab31fe58 100644 --- a/nvflare/tool/job/job_cli.py +++ b/nvflare/tool/job/job_cli.py @@ -41,13 +41,13 @@ JOB_CONFIG_FILE_NAME, JOB_CONFIG_VAR_NAME, JOB_CONFIG_VAR_VALUE, - JOB_INFO_CLIENT_TYPE, - JOB_INFO_CLIENT_TYPE_KEY, JOB_INFO_CONF, JOB_INFO_CONTROLLER_TYPE, JOB_INFO_CONTROLLER_TYPE_KEY, JOB_INFO_DESC, JOB_INFO_DESC_KEY, + JOB_INFO_EXECUTION_API_TYPE, + JOB_INFO_EXECUTION_API_TYPE_KEY, JOB_INFO_KEYS, JOB_INFO_MD, JOB_META_BASE_NAME, @@ -318,13 +318,13 @@ def display_available_templates(template_index_conf): print("-" * total_length) name_fix_length = 20 description_fix_length = 60 - controller_type_fix_length = 20 - client_category_fix_length = 20 + controller_type_fix_length = 17 + execution_api_type_fix_length = 23 name = fix_length_format("name", name_fix_length) description = fix_length_format(JOB_INFO_DESC, description_fix_length) - client_category = fix_length_format(JOB_INFO_CLIENT_TYPE, client_category_fix_length) + execution_api_type = fix_length_format(JOB_INFO_EXECUTION_API_TYPE, execution_api_type_fix_length) controller_type = fix_length_format(JOB_INFO_CONTROLLER_TYPE, controller_type_fix_length) - print(" " * left_margin, name, description, controller_type, client_category) + print(" " * left_margin, name, description, controller_type, execution_api_type) print("-" * total_length) for file_path in sorted(template_registry.keys()): name = os.path.basename(file_path) @@ -333,9 +333,11 @@ def display_available_templates(template_index_conf): template_info = template_registry.get(name) name = fix_length_format(name, name_fix_length) description = fix_length_format(template_info.get(JOB_INFO_DESC_KEY), description_fix_length) - client_category = fix_length_format(template_info.get(JOB_INFO_CLIENT_TYPE_KEY), client_category_fix_length) + execution_api_type = fix_length_format( + template_info.get(JOB_INFO_EXECUTION_API_TYPE_KEY), execution_api_type_fix_length + ) controller_type = fix_length_format(template_info.get(JOB_INFO_CONTROLLER_TYPE_KEY), controller_type_fix_length) - print(" " * left_margin, name, description, controller_type, client_category) + print(" " * left_margin, name, description, controller_type, execution_api_type) print("-" * total_length) diff --git a/nvflare/tool/job/job_client_const.py b/nvflare/tool/job/job_client_const.py index 8a26e64efc..8c37df5673 100644 --- a/nvflare/tool/job/job_client_const.py +++ b/nvflare/tool/job/job_client_const.py @@ -16,8 +16,8 @@ JOB_INFO_DESC = "Description" JOB_INFO_CONTROLLER_TYPE_KEY = "controller_type" JOB_INFO_CONTROLLER_TYPE = "Controller Type" -JOB_INFO_CLIENT_TYPE_KEY = "client_category" -JOB_INFO_CLIENT_TYPE = "Client Category" +JOB_INFO_EXECUTION_API_TYPE_KEY = "execution_api_type" +JOB_INFO_EXECUTION_API_TYPE = "Execution API Type" JOB_TEMPLATES = "job_templates" JOB_TEMPLATE = "job_template" @@ -25,7 +25,7 @@ JOB_INFO_CONF = "info.conf" JOB_INFO_MD = "info.md" -JOB_INFO_KEYS = [JOB_INFO_DESC_KEY, JOB_INFO_CONTROLLER_TYPE_KEY, JOB_INFO_CLIENT_TYPE_KEY] +JOB_INFO_KEYS = [JOB_INFO_DESC_KEY, JOB_INFO_CONTROLLER_TYPE_KEY, JOB_INFO_EXECUTION_API_TYPE_KEY] CONFIG_FILE_BASE_NAME_WO_EXTS = ["config_fed_client", "config_fed_server", "meta"] APP_CONFIG_FILE_BASE_NAMES = ["config_fed_client", "config_fed_server"] From 0b1ce6230e32934c0ce8741bd82c13f20c3a9a45 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Tue, 23 Jan 2024 17:37:22 -0800 Subject: [PATCH 14/39] Move workspace setup inside constructor (#2311) --- .../fed/app/simulator/simulator_runner.py | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/nvflare/private/fed/app/simulator/simulator_runner.py b/nvflare/private/fed/app/simulator/simulator_runner.py index 7e2d1cfd98..78fdc93e59 100644 --- a/nvflare/private/fed/app/simulator/simulator_runner.py +++ b/nvflare/private/fed/app/simulator/simulator_runner.py @@ -94,6 +94,15 @@ def __init__( self.clients_created = 0 + running_dir = os.getcwd() + if self.workspace is None: + self.workspace = "simulator_workspace" + self.logger.warn( + f"Simulator workspace is not provided. Set it to the default location:" + f" {os.path.join(running_dir, self.workspace)}" + ) + self.workspace = os.path.join(running_dir, self.workspace) + def _generate_args( self, job_folder: str, workspace: str, clients=None, n_clients=None, threads=None, gpu=None, max_clients=100 ): @@ -110,15 +119,6 @@ def _generate_args( return args def setup(self): - running_dir = os.getcwd() - if self.workspace is None: - self.workspace = "simulator_workspace" - self.logger.warn( - f"Simulator workspace is not provided. Set it to the default location:" - f" {os.path.join(running_dir, self.workspace)}" - ) - self.workspace = os.path.join(running_dir, self.workspace) - self.args = self._generate_args( self.job_folder, self.workspace, self.clients, self.n_clients, self.threads, self.gpu, self.max_clients ) @@ -348,7 +348,7 @@ def run(self): try: manager = Manager() return_dict = manager.dict() - process = Process(target=self.run_processs, args=(return_dict,)) + process = Process(target=self.run_process, args=(return_dict,)) process.start() process.join() run_status = self._get_return_code(return_dict, process, self.workspace) @@ -380,7 +380,7 @@ def _get_return_code(self, return_dict, process, workspace): self.logger.info(f"return_code from process.exitcode: {return_code}") return return_code - def run_processs(self, return_dict): + def run_process(self, return_dict): # run_status = self.simulator_run_main() try: run_status = mpm.run( From ab78879b10fe2cce1e4332cc6c0f48003fffab37 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Tue, 23 Jan 2024 20:07:32 -0800 Subject: [PATCH 15/39] Update cli string (#2313) --- nvflare/cli.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nvflare/cli.py b/nvflare/cli.py index eee979f7a7..46491cb130 100644 --- a/nvflare/cli.py +++ b/nvflare/cli.py @@ -48,7 +48,7 @@ def check_python_version(): if sys.version_info >= (3, 11): raise RuntimeError("Python versions 3.11 and above are not yet supported. Please use Python 3.8, 3.9 or 3.10.") if sys.version_info < (3, 8): - raise RuntimeError("Python versions 3.6 and below are not supported. Please use Python 3.8, 3.9 or 3.10") + raise RuntimeError("Python versions 3.7 and below are not supported. Please use Python 3.8, 3.9 or 3.10") def def_provision_parser(sub_cmd): From 06b59dfb69407d6381aeaf981fb93b410051d766 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Wed, 24 Jan 2024 10:56:02 -0800 Subject: [PATCH 16/39] Remove base class Filter (#2312) Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> --- nvflare/app_common/abstract/params_converter.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/nvflare/app_common/abstract/params_converter.py b/nvflare/app_common/abstract/params_converter.py index 6ec2e0e546..1ae611a836 100644 --- a/nvflare/app_common/abstract/params_converter.py +++ b/nvflare/app_common/abstract/params_converter.py @@ -16,12 +16,11 @@ from typing import Any, List from nvflare.apis.dxo import from_shareable -from nvflare.apis.filter import Filter from nvflare.apis.fl_context import FLContext from nvflare.apis.shareable import Shareable -class ParamsConverter(Filter, ABC): +class ParamsConverter(ABC): def __init__(self, supported_tasks: List[str] = None): self.supported_tasks = supported_tasks From 03af6b6ba0c2f9bf3145f492e773946a2aa88699 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Thu, 25 Jan 2024 09:59:20 -0800 Subject: [PATCH 17/39] Add hello-ccwf to ci (#2316) --- .../standalone_job/hello_numpy_examples.yml | 16 ++++++++++++++++ .../src/validators/np_sag_result_validator.py | 5 +++-- 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/tests/integration_test/data/test_configs/standalone_job/hello_numpy_examples.yml b/tests/integration_test/data/test_configs/standalone_job/hello_numpy_examples.yml index e578f70420..f3a87da6d7 100644 --- a/tests/integration_test/data/test_configs/standalone_job/hello_numpy_examples.yml +++ b/tests/integration_test/data/test_configs/standalone_job/hello_numpy_examples.yml @@ -43,3 +43,19 @@ tests: args: { server_model_names: ["server"] } - path: tests.integration_test.src.validators.NumpySAGResultValidator args: { expected_result: [ [ 4, 5, 6 ], [ 7, 8, 9 ], [ 10, 11, 12 ] ] } + - test_name: "run hello-ccwf" + # TODO: add a result validator for the "models" saved on client site (ccwf) + event_sequence: + - "trigger": + "type": "server_log" + "data": "Server started" + "actions": [ "submit_job hello-ccwf/jobs/swarm_cse_numpy" ] + "result": + "type": "job_submit_success" + - "trigger": + "type": "run_state" + "data": { "run_finished": True } + "actions": [ "ensure_current_job_done" ] + "result": + "type": "run_state" + "data": { "run_finished": True } diff --git a/tests/integration_test/src/validators/np_sag_result_validator.py b/tests/integration_test/src/validators/np_sag_result_validator.py index 0e8690abb1..bbf6c94d63 100644 --- a/tests/integration_test/src/validators/np_sag_result_validator.py +++ b/tests/integration_test/src/validators/np_sag_result_validator.py @@ -20,9 +20,10 @@ class NumpySAGResultValidator(FinishJobResultValidator): - def __init__(self, expected_result): + def __init__(self, expected_result, model_name: str = "server.npy"): super().__init__() self.expected_result = np.array(expected_result) + self.model_name = model_name def validate_finished_results(self, job_result, client_props) -> bool: server_run_dir = job_result["workspace_root"] @@ -32,7 +33,7 @@ def validate_finished_results(self, job_result, client_props) -> bool: self.logger.error(f"models dir {models_dir} doesn't exist.") return False - model_path = os.path.join(models_dir, "server.npy") + model_path = os.path.join(models_dir, self.model_name) if not os.path.isfile(model_path): self.logger.error(f"model_path {model_path} doesn't exist.") return False From c04785390840dba0b694379309f8c98fa49fb927 Mon Sep 17 00:00:00 2001 From: Sean Yang Date: Fri, 26 Jan 2024 09:34:13 -0800 Subject: [PATCH 18/39] Improve monai integration tracking example (#2318) * clarify monai integration tracking steps, remove print * add tracking_uri to config --- .../spleen_ct_segmentation_local/README.md | 60 +++++++-------- .../app/config/config_fed_client.json | 49 ------------ .../app/config/config_fed_server.json | 74 ------------------- .../meta.json | 10 --- .../app/config/config_fed_server.json | 5 +- .../spleen_ct_segmentation_sim/README.md | 9 ++- .../monai_nvflare/nvflare_stats_handler.py | 1 - 7 files changed, 40 insertions(+), 168 deletions(-) delete mode 100644 integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/app/config/config_fed_client.json delete mode 100644 integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/app/config/config_fed_server.json delete mode 100644 integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/meta.json diff --git a/integration/monai/examples/spleen_ct_segmentation_local/README.md b/integration/monai/examples/spleen_ct_segmentation_local/README.md index 7ba9892029..bd7d481316 100644 --- a/integration/monai/examples/spleen_ct_segmentation_local/README.md +++ b/integration/monai/examples/spleen_ct_segmentation_local/README.md @@ -21,8 +21,13 @@ And go to the folder containing this tutorial To execute the below commands, please open a terminal and go to the folder containing this tutorial. -We recommend following the instructions for setting up a [virtual environment](../../../../examples/README.md#set-up-a-virtual-environment), -and using it in [JupyterLab](../../../../examples/README.md#Set-up-JupyterLab-for-notebooks) for running the notebooks the MONAI integration examples. +Follow the [setup](../../README.md#requirements) to create a virtual environment with the MONAI-NVFlare integration installed to use in JupyterLab. + +Install the required packages in your virtual environment: + +``` +pip install -r ./requirements.txt +``` ### 1. Download the Spleen Bundle @@ -102,9 +107,9 @@ By default, POC will create startup kits at `/tmp/nvflare/poc`. ### 3.3 Start FL system in POC mode -Then, start the FL system with all provisioned clients by running +Then in another terminal start the FL system in POC mode with all provisioned clients by running: ``` -nvflare poc start +nvflare poc start -ex admin@nvidia.com ``` ### 4.1 (Optional) Secure FL workspace @@ -149,37 +154,26 @@ For details about resource management and consumption, please refer to the [docu > **Note:** Full FL training could take several hours for this task. > To speed up your experimentation, you can reduce the `num_rounds` value in `config_fed_server.json`, e.g. to 5 rounds. -### 5.1 FLARE-MONAI Integration Experiment tracking +### 5.1 FLARE-MONAI Integration Experiment Tracking with MLflow Experiment tracking for the FLARE-MONAI integration now uses `NVFlareStatsHandler` to provide a set of Ignite Event-handlers to support both iteration and epoch-level events for automatic metric streaming. -In this example, the `spleen_ct_segmentation_local` job is configured to automatically log metrics to MLflow through the FL server. - -The `config_fed_client.json` contains the `NVFlareStatsHandler`, `MetricsSender`, and `MetricRelay` (with their respective pipes) to send the metrics to the server side as federated events. -Then in `config_fed_server.json`, the `MLflowReceiver` is configured for the server to receive the results in "mlruns" or via the tracking uri if specified. - -View the results by running the following command at the `mlruns/` directory in the workspace: - -``` -mlflow ui --port 5000 -``` -> **_NOTE:_** The receiver on the server side can be easily configured to support other experiment tracking formats. -> In addition to the `MLflowReceiver`, the `WandBReceiver` and `TBAnalyticsReceiver` can also be used in `config_fed_server.json` for Tensorboard and > Weights & Biases experiment tracking streaming to the server. +In this example, the `spleen_ct_segmentation_local` job is configured to automatically log metrics to MLflow through the FL server. -### 5.2 MONAI Experiment tracking with MLflow +- The `config_fed_client.json` contains the `NVFlareStatsHandler`, `MetricsSender`, and `MetricRelay` (with their respective pipes) to send the metrics to the server side as federated events. +- Then in `config_fed_server.json`, the `MLflowReceiver` is configured for the server to write the results to the default MLflow tracking server URI. -The `spleen_ct_segmentation_loc_non_agg` job is the previous configuration that uses MONAI's experiment [tracking feature](https://github.com/Project-MONAI/tutorials/tree/main/experiment_management) -with clients logging to the MLflow tracking server without going through the FL server. -For `spleen_ct_segmentation_loc_non_agg`, an MLflow server is expected, so in a new terminal, start the mlflow server with: +With this configuration the MLflow tracking server must be started before running the job: ``` mlflow server ``` -You can access the MLflow dashboard in your browser using the default tracking uri `http://127.0.0.1:5000`. +> **_NOTE:_** The receiver on the server side can be easily configured to support other experiment tracking formats. + In addition to the `MLflowReceiver`, the `WandBReceiver` and `TBAnalyticsReceiver` can also be used in `config_fed_server.json` for Tensorboard and Weights & Biases experiment tracking streaming to the server. -Next, submit the job. +Next, we can submit the job. -### 5.3 Federated averaging +### 5.2 Federated averaging To run FedAvg using with the Job CLI, submit the job with: @@ -207,13 +201,9 @@ You should see the cross-site validation results at [DOWNLOAD_DIR]/[JOB_ID]/workspace/cross_site_val/cross_val_results.json ``` -Once the training started, you can the experiment curves for the local clients in the current run on the MLflow dashboard. - -![MLflow dashboard](./mlflow.png) - -### 5.4 Secure aggregation using homomorphic encryption +### 5.3 Secure aggregation using homomorphic encryption -Next we run FedAvg using homomorphic encryption (HE) for secure aggregation on the server. +Alternatively we can run FedAvg using homomorphic encryption (HE) for secure aggregation on the server. > **_NOTE:_** For HE, we need to use the securely provisioned workspace. > It will also take longer due to the additional encryption, decryption, encrypted aggregation, @@ -225,3 +215,13 @@ Then, submit the job to run FedAvg with HE: ``` nvflare job submit -j jobs/spleen_ct_segementation_he ``` + +### 5.4 MLflow experiment tracking results + +To view the results, you can access the MLflow dashboard in your browser using the default tracking uri `http://127.0.0.1:5000`. + +> **_NOTE:_** To write the results to the server workspace instead of using the MLflow server, users can remove the `tracking_uri` argument from the `MLflowReceiver` configuration and instead view the results by running `mlflow ui --port 5000` in the directory that contains the `mlruns/` directory in the server workspace. + +Once the training is started, you can see the experiment curves for the local clients in the current run on the MLflow dashboard. + +![MLflow dashboard](./mlflow.png) \ No newline at end of file diff --git a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/app/config/config_fed_client.json b/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/app/config/config_fed_client.json deleted file mode 100644 index a8bd20e94f..0000000000 --- a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/app/config/config_fed_client.json +++ /dev/null @@ -1,49 +0,0 @@ -{ - "format_version": 2, - - "executors": [ - { - "tasks": [ - "train", "submit_model", "validate" - ], - "executor": { - "id": "executor", - "path": "monai_nvflare.client_algo_executor.ClientAlgoExecutor", - "args": { - "client_algo_id": "client_algo", - "key_metric": "val_mean_dice" - } - } - } - ], - - "task_result_filters": [ - ], - "task_data_filters": [ - ], - - "tracking": "mlflow", - "experiment_name": "monai_nvflare", - "tracking_uri": "http://127.0.0.1:5000", - - "components": [ - { - "id": "client_algo", - "path": "monai.fl.client.MonaiAlgo", - "args": { - "bundle_root": "config/spleen_ct_segmentation", - "local_epochs": 10, - "train_kwargs": { - "tracking": "{tracking}", - "tracking_uri": "{tracking_uri}", - "experiment_name": "{experiment_name}" - }, - "eval_kwargs": { - "tracking": "{tracking}", - "tracking_uri": "{tracking_uri}", - "experiment_name": "{experiment_name}" - } - } - } - ] -} diff --git a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/app/config/config_fed_server.json b/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/app/config/config_fed_server.json deleted file mode 100644 index 581e3a8c26..0000000000 --- a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/app/config/config_fed_server.json +++ /dev/null @@ -1,74 +0,0 @@ -{ - "format_version": 2, - - "min_clients": 2, - "num_rounds": 100, - - "task_data_filters": [], - "task_result_filters": [], - "components": [ - { - "id": "persistor", - "path": "monai_nvflare.monai_bundle_persistor.MonaiBundlePersistor", - "args": { - "bundle_root": "config/spleen_ct_segmentation" - } - }, - { - "id": "shareable_generator", - "name": "FullModelShareableGenerator", - "args": {} - }, - { - "id": "aggregator", - "name": "InTimeAccumulateWeightedAggregator", - "args": { - "expected_data_kind": "WEIGHT_DIFF" - } - }, - { - "id": "model_selector", - "name": "IntimeModelSelector", - "args": {} - }, - { - "id": "model_locator", - "name": "PTFileModelLocator", - "args": { - "pt_persistor_id": "persistor" - } - }, - { - "id": "json_generator", - "name": "ValidationJsonGenerator", - "args": {} - } - ], - "workflows": [ - { - "id": "scatter_gather_ctl", - "name": "ScatterAndGather", - "args": { - "min_clients" : "{min_clients}", - "num_rounds" : "{num_rounds}", - "start_round": 0, - "wait_time_after_min_received": 10, - "aggregator_id": "aggregator", - "persistor_id": "persistor", - "shareable_generator_id": "shareable_generator", - "train_task_name": "train", - "train_timeout": 0 - } - }, - { - "id": "cross_site_model_eval", - "name": "CrossSiteModelEval", - "args": { - "model_locator_id": "model_locator", - "submit_model_timeout": 600, - "validation_timeout": 6000, - "cleanup_models": true - } - } - ] -} diff --git a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/meta.json b/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/meta.json deleted file mode 100644 index 4947562644..0000000000 --- a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_loc_non_agg/meta.json +++ /dev/null @@ -1,10 +0,0 @@ -{ - "name": "spleen-bundle", - "resource_spec": {}, - "min_clients" : 2, - "deploy_map": { - "app": [ - "@ALL" - ] - } -} diff --git a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.json b/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.json index dae4608749..749b59deb5 100644 --- a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.json +++ b/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.json @@ -47,14 +47,15 @@ "id": "mlflow_receiver_with_tracking_uri", "path": "nvflare.app_opt.tracking.mlflow.mlflow_receiver.MLflowReceiver", "args": { + "tracking_uri": "http://127.0.0.1:5000", "kwargs": { "experiment_name": "monai-spleen-experiment", "run_name": "monai-spleen-with-mlflow", "experiment_tags": { - "mlflow.note.content": "## **MONAI experiment with spleen bundle with MLflow**" + "mlflow-note-content": "## **MONAI experiment with spleen bundle with MLflow**" }, "run_tags": { - "mlflow.note.content": "## Federated Experiment tracking with MLflow \n### Example of using **[NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html)** to train and run MONAI-bundle using federated averaging ([FedAvg]([FedAvg](https://arxiv.org/abs/1602.05629))) and [PyTorch](https://pytorch.org/) as the deep learning training framework. This example also highlights the FLARE streaming capability from the clients to the server for server delivery to MLflow.\n" + "mlflow-note-content": "## Federated Experiment tracking with MLflow \n### Example of using **[NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html)** to train and run MONAI-bundle using federated averaging ([FedAvg]([FedAvg](https://arxiv.org/abs/1602.05629))) and [PyTorch](https://pytorch.org/) as the deep learning training framework. This example also highlights the FLARE streaming capability from the clients to the server for server delivery to MLflow.\n" } }, "artifact_location": "artifacts" diff --git a/integration/monai/examples/spleen_ct_segmentation_sim/README.md b/integration/monai/examples/spleen_ct_segmentation_sim/README.md index 5c951bde8f..e7b8dd03ed 100644 --- a/integration/monai/examples/spleen_ct_segmentation_sim/README.md +++ b/integration/monai/examples/spleen_ct_segmentation_sim/README.md @@ -9,8 +9,13 @@ For an example with real-world deployment settings, see [here](../spleen_ct_segm To execute the below commands, please open a terminal and go to the folder containing this tutorial. -We recommend following the instructions for setting up a [virtual environment](../../../../examples/README.md#set-up-a-virtual-environment), -and using it in [JupyterLab](../../../../examples/README.md#Set-up-JupyterLab-for-notebooks) for running the notebooks the MONAI integration examples. +Follow the [setup](../../README.md#requirements) to create a virtual environment with the MONAI-NVFlare integration installed to use in JupyterLab. + +Install the required packages in your virtual environment: + +``` +pip install -r ./requirements.txt +``` ### 1. Download the Spleen Bundle diff --git a/integration/monai/monai_nvflare/nvflare_stats_handler.py b/integration/monai/monai_nvflare/nvflare_stats_handler.py index 07d5204bd5..2a7ab7d840 100644 --- a/integration/monai/monai_nvflare/nvflare_stats_handler.py +++ b/integration/monai/monai_nvflare/nvflare_stats_handler.py @@ -171,7 +171,6 @@ def _default_epoch_sender(self, engine: Engine) -> None: current_epoch = self.global_epoch_transform(engine.state.epoch) summary_dict = engine.state.metrics for name, value in summary_dict.items(): - print(f"\n\t{name}", type(value), value) self._send_stats(engine, name, value, AnalyticsDataType.SCALAR, current_epoch) if self.state_attributes is not None: From 260c38c91798578428abe58adcace5758e1fd8de Mon Sep 17 00:00:00 2001 From: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Date: Mon, 29 Jan 2024 14:47:33 -0800 Subject: [PATCH 19/39] Exclude hidden json file when perform config parsing (#2323) * add Clara FL white paper * exclude nvfl hidden json files when parsing * address PR comment --- nvflare/lighter/tool_consts.py | 17 +++++++++++++++++ nvflare/lighter/utils.py | 13 +++++++------ nvflare/tool/job/config/configer.py | 3 ++- nvflare/tool/poc/poc_commands.py | 6 +++--- 4 files changed, 29 insertions(+), 10 deletions(-) create mode 100644 nvflare/lighter/tool_consts.py diff --git a/nvflare/lighter/tool_consts.py b/nvflare/lighter/tool_consts.py new file mode 100644 index 0000000000..2ba1970a9f --- /dev/null +++ b/nvflare/lighter/tool_consts.py @@ -0,0 +1,17 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +NVFLARE_PREFIX = ".__nvfl_" +NVFLARE_SIG_FILE = ".__nvfl_sig.json" +NVFLARE_SUBMITTER_CRT_FILE = ".__nvfl_submitter.crt" diff --git a/nvflare/lighter/utils.py b/nvflare/lighter/utils.py index e7836537d8..2bb1b2b753 100644 --- a/nvflare/lighter/utils.py +++ b/nvflare/lighter/utils.py @@ -24,6 +24,7 @@ from cryptography.hazmat.primitives.asymmetric import padding from nvflare.lighter.impl.cert import load_crt +from nvflare.lighter.tool_consts import NVFLARE_SIG_FILE, NVFLARE_SUBMITTER_CRT_FILE def generate_password(passlen=16): @@ -56,7 +57,7 @@ def sign_folders(folder, signing_pri_key, crt_path, max_depth=9999): depth = depth + 1 signatures = dict() for file in files: - if file == ".__nvfl_sig.json" or file == ".__nvfl_submitter.crt": + if file == NVFLARE_SIG_FILE or file == NVFLARE_SUBMITTER_CRT_FILE: continue signature = signing_pri_key.sign( data=open(os.path.join(root, file), "rb").read(), @@ -78,8 +79,8 @@ def sign_folders(folder, signing_pri_key, crt_path, max_depth=9999): ) signatures[folder] = b64encode(signature).decode("utf-8") - json.dump(signatures, open(os.path.join(root, ".__nvfl_sig.json"), "wt")) - shutil.copyfile(crt_path, os.path.join(root, ".__nvfl_submitter.crt")) + json.dump(signatures, open(os.path.join(root, NVFLARE_SIG_FILE), "wt")) + shutil.copyfile(crt_path, os.path.join(root, NVFLARE_SUBMITTER_CRT_FILE)) if depth >= max_depth: break @@ -90,8 +91,8 @@ def verify_folder_signature(src_folder, root_ca_path): root_ca_public_key = root_ca_cert.public_key() for root, folders, files in os.walk(src_folder): try: - signatures = json.load(open(os.path.join(root, ".__nvfl_sig.json"), "rt")) - cert = load_crt(os.path.join(root, ".__nvfl_submitter.crt")) + signatures = json.load(open(os.path.join(root, NVFLARE_SIG_FILE), "rt")) + cert = load_crt(os.path.join(root, NVFLARE_SUBMITTER_CRT_FILE)) public_key = cert.public_key() except: continue # TODO: shall return False @@ -101,7 +102,7 @@ def verify_folder_signature(src_folder, root_ca_path): for k in signatures: signatures[k] = b64decode(signatures[k].encode("utf-8")) for file in files: - if file == ".__nvfl_sig.json" or file == ".__nvfl_submitter.crt": + if file == NVFLARE_SIG_FILE or file == NVFLARE_SUBMITTER_CRT_FILE: continue signature = signatures.get(file) if signature: diff --git a/nvflare/tool/job/config/configer.py b/nvflare/tool/job/config/configer.py index 0221ac0b3a..e50dafc254 100644 --- a/nvflare/tool/job/config/configer.py +++ b/nvflare/tool/job/config/configer.py @@ -18,6 +18,7 @@ from pyhocon import ConfigFactory, ConfigTree from nvflare.fuel.utils.config import ConfigFormat +from nvflare.lighter.tool_consts import NVFLARE_PREFIX from nvflare.tool.job.config.config_indexer import KeyIndex, build_reverse_order_index from nvflare.tool.job.job_client_const import ( APP_CONFIG_DIR, @@ -436,7 +437,7 @@ def build_config_file_indices(job_folder: str, app_names: List[str]) -> Dict[str for root, dirs, files in os.walk(custom_dir): for f in files: for ext in config_extensions: - if f.endswith(ext): + if f.endswith(ext) and not f.startswith(NVFLARE_PREFIX): file = os.path.join(root, f) config_files = app_config_files.get(app_name, []) config_files.append(file) diff --git a/nvflare/tool/poc/poc_commands.py b/nvflare/tool/poc/poc_commands.py index 7809e64cd8..d1fcc5b274 100644 --- a/nvflare/tool/poc/poc_commands.py +++ b/nvflare/tool/poc/poc_commands.py @@ -164,19 +164,19 @@ def _prepare_jobs_dir(jobs_dir: str, workspace: str, config_packages: Optional[T dst = os.path.join(console_dir, transfer) if not is_dir_empty(dst): print(" ") - answer = input(f"Examples at {dst} is already exists, replace with new one ? (y/N) ") + answer = input(f"job directory at {dst} is already exists, replace with new one ? (y/N) ") if answer.strip().upper() == "Y": if os.path.islink(dst): os.unlink(dst) if os.path.isdir(dst): shutil.rmtree(dst, ignore_errors=True) - print(f"link examples from {src} to {dst}") + print(f"link job directory from {src} to {dst}") os.symlink(src, dst) else: if os.path.isdir(dst): shutil.rmtree(dst, ignore_errors=True) - print(f"link examples from {src} to {dst}") + print(f"link job directory from {src} to {dst}") os.symlink(src, dst) From 3589df10b49e29f10ebe3bd8d5e9a3d6279f87d7 Mon Sep 17 00:00:00 2001 From: Holger Roth <6304754+holgerroth@users.noreply.github.com> Date: Mon, 29 Jan 2024 18:34:36 -0500 Subject: [PATCH 20/39] apply nemo 2.4 fixes (#2327) --- integration/nemo/README.md | 59 +-------- integration/nemo/examples/README.md | 12 +- integration/nemo/examples/peft/README.md | 24 +++- .../examples/peft/nemo_nvflare/__init__.py | 15 +++ .../megatron_gpt_peft_fl_eval_config.yaml | 0 .../megatron_gpt_peft_tuning.py | 0 .../megatron_gpt_peft_tuning_config.yaml | 0 .../peft}/nemo_nvflare/peft_model.py | 0 .../nemo/examples/peft/nemo_nvflare/utils.py | 34 +++++ integration/nemo/examples/peft/peft.ipynb | 116 +++++++++--------- .../nemo/examples/prompt_learning/README.md | 31 +++-- .../prompt_learning}/nemo_nvflare/__init__.py | 4 - .../nemo_nvflare/config_sharer.py | 0 .../nemo_nvflare/constants.py | 0 .../fed_megatron_gpt_prompt_learning_model.py | 0 .../nemo_nvflare/learner_executor.py | 0 .../nemo_nvflare/prompt_encoder.py | 0 .../nemo_nvflare/prompt_learner.py | 0 .../nemo_nvflare/share_config.py | 0 .../prompt_learning}/nemo_nvflare/utils.py | 0 .../prompt_learning/prompt_learning.ipynb | 100 +++++++-------- .../prompt_learning/prompt_learning_20B.md | 29 +++-- .../examples/supervised_fine_tuning/README.md | 67 ++++++---- .../nemo_nvflare/__init__.py | 19 +++ .../nemo_nvflare/config_sharer_sft.py | 0 .../nemo_nvflare/constants.py | 23 ++++ .../nemo_nvflare/learner_executor.py | 80 ++++++++++++ .../nemo_nvflare/server_sft_model.py | 0 .../nemo_nvflare/sft_learner.py | 0 .../nemo_nvflare/share_config_sft.py | 0 .../nemo_nvflare/utils_sft.py | 0 31 files changed, 400 insertions(+), 213 deletions(-) create mode 100644 integration/nemo/examples/peft/nemo_nvflare/__init__.py rename integration/nemo/examples/peft/{code => nemo_nvflare}/megatron_gpt_peft_fl_eval_config.yaml (100%) rename integration/nemo/examples/peft/{code => nemo_nvflare}/megatron_gpt_peft_tuning.py (100%) rename integration/nemo/examples/peft/{code => nemo_nvflare}/megatron_gpt_peft_tuning_config.yaml (100%) rename integration/nemo/{ => examples/peft}/nemo_nvflare/peft_model.py (100%) create mode 100644 integration/nemo/examples/peft/nemo_nvflare/utils.py rename integration/nemo/{ => examples/prompt_learning}/nemo_nvflare/__init__.py (84%) rename integration/nemo/{ => examples/prompt_learning}/nemo_nvflare/config_sharer.py (100%) rename integration/nemo/{ => examples/prompt_learning}/nemo_nvflare/constants.py (100%) rename integration/nemo/{ => examples/prompt_learning}/nemo_nvflare/fed_megatron_gpt_prompt_learning_model.py (100%) rename integration/nemo/{ => examples/prompt_learning}/nemo_nvflare/learner_executor.py (100%) rename integration/nemo/{ => examples/prompt_learning}/nemo_nvflare/prompt_encoder.py (100%) rename integration/nemo/{ => examples/prompt_learning}/nemo_nvflare/prompt_learner.py (100%) rename integration/nemo/{ => examples/prompt_learning}/nemo_nvflare/share_config.py (100%) rename integration/nemo/{ => examples/prompt_learning}/nemo_nvflare/utils.py (100%) create mode 100644 integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/__init__.py rename integration/nemo/{ => examples/supervised_fine_tuning}/nemo_nvflare/config_sharer_sft.py (100%) create mode 100644 integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/constants.py create mode 100644 integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/learner_executor.py rename integration/nemo/{ => examples/supervised_fine_tuning}/nemo_nvflare/server_sft_model.py (100%) rename integration/nemo/{ => examples/supervised_fine_tuning}/nemo_nvflare/sft_learner.py (100%) rename integration/nemo/{ => examples/supervised_fine_tuning}/nemo_nvflare/share_config_sft.py (100%) rename integration/nemo/{ => examples/supervised_fine_tuning}/nemo_nvflare/utils_sft.py (100%) diff --git a/integration/nemo/README.md b/integration/nemo/README.md index 94a1dd4e7e..4cb4ed604b 100644 --- a/integration/nemo/README.md +++ b/integration/nemo/README.md @@ -1,63 +1,16 @@ # NeMo Integration -## Objective -Execute [NVIDIA NeMo™](https://developer.nvidia.com/nemo) in federated environments. - -### Goals: - -Allow NeMo models to be trained and adapted with NVFlare. - -### Non-goals: - -n/a - -## Background -NVIDIA NeMo™ is an end-to-end cloud-native enterprise framework for developers to +[NVIDIA NeMo™](https://developer.nvidia.com/nemo) is an end-to-end cloud-native enterprise framework for developers to build, customize, and deploy generative AI models with billions of parameters. -## Description -NVFlare utilizes features from NeMo, such as prompt learning to run LLM tasks in federated environments. - -### Examples - -For an example of using [NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html) with NeMo for prompt learning, -see [examples/prompt_learning](examples/prompt_learning/README.md) - -For an example of using [NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html) with NeMo for supervised fine-tuning (SFT), -see [examples/supervised_fine_tuning](examples/supervised_fine_tuning/README.md) +Here, we show how NVFlare utilizes features from NeMo to run LLM tasks in federated environments with several [examples](./examples). ## Requirements -### Using docker -For simplicity, we recommend using NVIDIA's docker containers that include all the requirements for running NeMo models. -``` -docker pull nvcr.io/nvidia/nemo:23.02 -``` - -### Install NeMo-NVFlare package - - - -#### Mount the source code -For easy development with NeMo, install NVFlare and mount the code inside the folder. -``` -pip install nvflare>=2.3.0 -export PYTHONPATH=${PWD} -``` +### Using docker (Recommended) +For simplicity, we recommend using NVIDIA's [NeMo docker containers](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo) that include all the requirements for running NeMo models. - +> Note: each example in this folder might require different container version. Please check their Readmes for details. ### Installation in a virtual environment @@ -68,4 +21,4 @@ and using it in [JupyterLab](../../examples/README.md#notebooks) for running the notebooks in the NeMo integration examples. Follow the NeMo installation steps [here](https://github.com/NVIDIA/NeMo#installation) -before installing the NeMo-NVFlare package. +before installing NVFlare and adding the source to the PYTHONPATH. diff --git a/integration/nemo/examples/README.md b/integration/nemo/examples/README.md index 4e7ed42f32..7551091184 100644 --- a/integration/nemo/examples/README.md +++ b/integration/nemo/examples/README.md @@ -1,16 +1,16 @@ # Examples of NeMo-NVFlare Integration ### [Parameter-Efficient Fine-Tuning (PEFT) with NeMo](./peft/README.md) -In this example, we utilize NeMo's [PEFT](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/nemo_megatron/peft/landing_page.html) +In this example, we utilize NeMo's [PEFT](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/nlp/nemo_megatron/peft/landing_page.html) using NVFlare's new Client API (minimal code changes required to run a NeMo script in FL) methods to showcase how to adapt a large language model (LLM) to a downstream task, such as financial sentiment predictions. -### [Prompt learning with NeMo and NVFlare](./prompt_learning/README.md) -An example of using [NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html) -with NeMo for [prompt learning](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/prompt_learning.html) -to adapt a large language model (LLM) to a downstream task. - ### [Supervised fine-tuning (SFT) with NeMo and NVFlare](./prompt_learning/README.md) An example of using [NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html) with NeMo for [supervised fine-tuning (SFT)](https://github.com/NVIDIA/NeMo-Megatron-Launcher#5152-sft-training) to fine-tune all parameters of a large language model (LLM) on supervised data to teach the model how to follow user specified instructions. + +### [Prompt learning with NeMo and NVFlare](./prompt_learning/README.md) +An example of using [NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html) +with NeMo for [prompt learning](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/prompt_learning.html) using NVFlare's Learner API +to adapt a large language model (LLM) to a downstream task. diff --git a/integration/nemo/examples/peft/README.md b/integration/nemo/examples/peft/README.md index c5f1085cbb..bab36e0487 100644 --- a/integration/nemo/examples/peft/README.md +++ b/integration/nemo/examples/peft/README.md @@ -10,17 +10,33 @@ that condition the model to produce the desired output for the downstream task. For more details, see the [PEFT script](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/tuning/megatron_gpt_peft_tuning.py) in NeMo, which we adapt using NVFlare's Lightning client API to run in a federated scenario. ## Dependencies -We assume you followed the instructions [here](../../README.md#requirements) -to install the NeMo, NVFlare, and the NeMo-NVFlare package. +The example was tested with the [NeMo 23.10 container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo). +In the following, we assume this example folder of the container is mounted to `/workspace` and all downloading, etc. operations are based on this root path. -The example was tested with the main branch of [NeMo](https://github.com/NVIDIA/NeMo). +> Note in the following, mount both the [current directory](./) and the [job_templates](../../../../job_templates) +> directory to locations inside the docker container. Please make sure you have cloned the full NVFlare repo. + +Start the docker container using +``` +DOCKER_IMAGE="nvcr.io/nvidia/nemo:23.10" +docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 \ +-v ${PWD}/../../../../job_templates:/job_templates -v ${PWD}:/workspace -w /workspace ${DOCKER_IMAGE} +``` + +For easy experimentation with NeMo, install NVFlare and mount the code inside the [nemo_nvflare](./nemo_nvflare) folder. +``` +pip install nvflare~=2.4.0rc7 +export PYTHONPATH=${PYTHONPATH}:/workspace +``` ## Examples ### 1. Federated PEFT using a 345 million parameter GPT model -This example requires a GPU with at least 24GB memory to run three clients in parallel on the same GPU. We use [JupyterLab](https://jupyterlab.readthedocs.io) for this example. To start JupyterLab, run ``` jupyter lab . ``` and open [peft.ipynb](./peft.ipynb). + +#### Hardware requirement +This example requires a GPU with at least 24GB memory to run three clients in parallel on the same GPU. diff --git a/integration/nemo/examples/peft/nemo_nvflare/__init__.py b/integration/nemo/examples/peft/nemo_nvflare/__init__.py new file mode 100644 index 0000000000..d6050992d1 --- /dev/null +++ b/integration/nemo/examples/peft/nemo_nvflare/__init__.py @@ -0,0 +1,15 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .peft_model import PEFTmodel diff --git a/integration/nemo/examples/peft/code/megatron_gpt_peft_fl_eval_config.yaml b/integration/nemo/examples/peft/nemo_nvflare/megatron_gpt_peft_fl_eval_config.yaml similarity index 100% rename from integration/nemo/examples/peft/code/megatron_gpt_peft_fl_eval_config.yaml rename to integration/nemo/examples/peft/nemo_nvflare/megatron_gpt_peft_fl_eval_config.yaml diff --git a/integration/nemo/examples/peft/code/megatron_gpt_peft_tuning.py b/integration/nemo/examples/peft/nemo_nvflare/megatron_gpt_peft_tuning.py similarity index 100% rename from integration/nemo/examples/peft/code/megatron_gpt_peft_tuning.py rename to integration/nemo/examples/peft/nemo_nvflare/megatron_gpt_peft_tuning.py diff --git a/integration/nemo/examples/peft/code/megatron_gpt_peft_tuning_config.yaml b/integration/nemo/examples/peft/nemo_nvflare/megatron_gpt_peft_tuning_config.yaml similarity index 100% rename from integration/nemo/examples/peft/code/megatron_gpt_peft_tuning_config.yaml rename to integration/nemo/examples/peft/nemo_nvflare/megatron_gpt_peft_tuning_config.yaml diff --git a/integration/nemo/nemo_nvflare/peft_model.py b/integration/nemo/examples/peft/nemo_nvflare/peft_model.py similarity index 100% rename from integration/nemo/nemo_nvflare/peft_model.py rename to integration/nemo/examples/peft/nemo_nvflare/peft_model.py diff --git a/integration/nemo/examples/peft/nemo_nvflare/utils.py b/integration/nemo/examples/peft/nemo_nvflare/utils.py new file mode 100644 index 0000000000..7ca186eae5 --- /dev/null +++ b/integration/nemo/examples/peft/nemo_nvflare/utils.py @@ -0,0 +1,34 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import numpy as np +import torch + + +def convert_global_to_ckpt(global_model_filepath: str, ckpt_path: str): + """Helper function to convert global models saved by NVFlare to NeMo ckpt format""" + + nvflare_ckpt = torch.load(global_model_filepath) + if "train_conf" in nvflare_ckpt: + print("Loaded NVFlare global checkpoint with train_conf", nvflare_ckpt["train_conf"]) + + assert ( + "model" in nvflare_ckpt + ), f"Expected global model to contain a 'model' key but it only had {list(nvflare_ckpt.keys())}" + global_weights = nvflare_ckpt["model"] + + torch.save({"state_dict": global_weights}, ckpt_path) + + print(f"Saved NeMo ckpt with {len(global_weights)} entries to {ckpt_path}") + diff --git a/integration/nemo/examples/peft/peft.ipynb b/integration/nemo/examples/peft/peft.ipynb index a86d28b4b5..ba31f585b9 100644 --- a/integration/nemo/examples/peft/peft.ipynb +++ b/integration/nemo/examples/peft/peft.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "0c534975", + "id": "5020dd81", "metadata": {}, "source": [ "# Parameter-Efficient Fine-Tuning (PEFT) with NeMo\n", @@ -19,17 +19,17 @@ }, { "cell_type": "markdown", - "id": "513eb148", + "id": "dc9769ef", "metadata": {}, "source": [ "## Dependencies\n", - "We assume you followed the instructions [here](../../README.md#requirements) \n", - "to install the NeMo framework and the NeMo-NVFlare package. " + "We assume you followed the instructions [here](./README.md) \n", + "to install the NeMo and NVFlare frameworks and mount the required codes." ] }, { "cell_type": "markdown", - "id": "bb97927a", + "id": "dab4c639", "metadata": {}, "source": [ "## Download the pre-trained LLM\n", @@ -39,7 +39,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c2f6c8b5", + "id": "20921eea", "metadata": {}, "outputs": [], "source": [ @@ -51,7 +51,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2036e09e", + "id": "aa852d07", "metadata": {}, "outputs": [], "source": [ @@ -66,7 +66,7 @@ }, { "cell_type": "markdown", - "id": "67f48638", + "id": "fa530a42", "metadata": {}, "source": [ "## Data preprocessing\n", @@ -74,12 +74,12 @@ "\n", "The Financial PhraseBank dataset contains the sentiments for financial news headlines from a retail investor's perspective. Further details about the dataset can be found in Malo et al.'s [\"Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts\"](https://arxiv.org/abs/1307.5336).\n", "\n", - "We can configure the prompt template used by NeMo to solve this downstream task by setting `prompt_template: \"{sentence} sentiment: {label}\"` in [megatron_gpt_peft_tuning_config.yaml](./code/megatron_gpt_peft_tuning_config.yaml) accordingly." + "We can configure the prompt template used by NeMo to solve this downstream task by setting `prompt_template: \"{sentence} sentiment: {label}\"` in [megatron_gpt_peft_tuning_config.yaml](./nemo_nvflare/megatron_gpt_peft_tuning_config.yaml) accordingly." ] }, { "cell_type": "markdown", - "id": "29dd0470", + "id": "b5737e50", "metadata": {}, "source": [ "#### 1. Download the preprocessing scripts\n", @@ -89,7 +89,7 @@ { "cell_type": "code", "execution_count": null, - "id": "f37039ed", + "id": "b2c32fa5", "metadata": {}, "outputs": [], "source": [ @@ -102,7 +102,7 @@ }, { "cell_type": "markdown", - "id": "f1b1a07b", + "id": "13a2f952", "metadata": {}, "source": [ "#### 2. Download the Financial PhraseBank Dataset\n", @@ -114,7 +114,7 @@ }, { "cell_type": "markdown", - "id": "f335899e", + "id": "40199807", "metadata": {}, "source": [ "#### 3. Preprocess the dataset" @@ -123,7 +123,7 @@ { "cell_type": "code", "execution_count": null, - "id": "dc66ef42", + "id": "80f84586", "metadata": {}, "outputs": [], "source": [ @@ -132,7 +132,7 @@ }, { "cell_type": "markdown", - "id": "365a58c8", + "id": "d9f8fa9a", "metadata": {}, "source": [ "#### 4. Split the dataset to simulate clients\n", @@ -143,7 +143,7 @@ { "cell_type": "code", "execution_count": null, - "id": "3f9214af", + "id": "a6725683", "metadata": {}, "outputs": [], "source": [ @@ -160,7 +160,7 @@ }, { "cell_type": "markdown", - "id": "cc85565d", + "id": "6c506c6b", "metadata": {}, "source": [ "Below are some examples of how the training data is distributed amount the three clients when using different values of `alpha`.\n", @@ -173,7 +173,7 @@ }, { "cell_type": "markdown", - "id": "3eea187a", + "id": "704ff05d", "metadata": {}, "source": [ "## Federated learning simulations\n", @@ -187,7 +187,7 @@ }, { "cell_type": "markdown", - "id": "aa23b7c7", + "id": "01fce4ae", "metadata": {}, "source": [ "#### 1. Convert NeMo PEFT script to FL\n", @@ -205,7 +205,7 @@ "\"Drawing\"\n", "\n", "\n", - "You can directly use all the PEFT methods implemented in the NeMo script, by changing the value of [peft_scheme](./code/megatron_gpt_peft_tuning_config.yaml) in the client configuration shown below accordingly:\n", + "You can directly use all the PEFT methods implemented in the NeMo script, by changing the value of [peft_scheme](./nemo_nvflare/megatron_gpt_peft_tuning_config.yaml) in the client configuration shown below accordingly:\n", "* p-tuning\n", "* adapter + p-tuning\n", "* adapter\n", @@ -221,7 +221,7 @@ }, { "cell_type": "markdown", - "id": "95b07067", + "id": "655a1f0a", "metadata": {}, "source": [ "#### 1. Local training\n", @@ -236,16 +236,16 @@ { "cell_type": "code", "execution_count": null, - "id": "b6e001c1", + "id": "51e4fb4d", "metadata": {}, "outputs": [], "source": [ - "!nvflare config -jt ../../../../job_templates" + "!nvflare config -jt /job_templates" ] }, { "cell_type": "markdown", - "id": "f3528af2", + "id": "2e515dc2", "metadata": {}, "source": [ "Then, create the job and configure it for simulating local training." @@ -254,8 +254,10 @@ { "cell_type": "code", "execution_count": null, - "id": "9905ebaa", - "metadata": {}, + "id": "404fe5fe", + "metadata": { + "scrolled": true + }, "outputs": [], "source": [ "import os\n", @@ -272,7 +274,7 @@ "num_rounds=1\n", "trainer_config=\"trainer.max_steps\\=1000 trainer.val_check_interval\\=100\"\n", "\n", - "!nvflare job create -force -j \"./jobs/peft_{peft_scheme}_local_345M\" -w \"sag_nemo\" -sd \"code\" \\\n", + "!nvflare job create -force -j \"./jobs/peft_{peft_scheme}_local_345M\" -w \"sag_nemo\" -sd \"nemo_nvflare\" \\\n", " -f app_1/config_fed_client.conf app_script={app_script} app_config=\"{peft_scheme_arg} model.restore_from_path\\={restore_from_path} {trainer_config} {val_files} {train_files_prefix}-1.jsonl\\]\" \\\n", " -f app_2/config_fed_client.conf app_script={app_script} app_config=\"{peft_scheme_arg} model.restore_from_path\\={restore_from_path} {trainer_config} {val_files} {train_files_prefix}-2.jsonl\\]\" \\\n", " -f app_3/config_fed_client.conf app_script={app_script} app_config=\"{peft_scheme_arg} model.restore_from_path\\={restore_from_path} {trainer_config} {val_files} {train_files_prefix}-3.jsonl\\]\" \\\n", @@ -281,7 +283,7 @@ }, { "cell_type": "markdown", - "id": "945b7d71", + "id": "df9ca0a5", "metadata": {}, "source": [ "Next, simulate each client training on their local dataset using the FL simulator. To do this, we only run 1 round of FL, with each client running 1000 steps on their local dataset." @@ -290,12 +292,16 @@ { "cell_type": "code", "execution_count": null, - "id": "09ef104c", + "id": "8d7f4970", "metadata": { "scrolled": true }, "outputs": [], "source": [ + "# required by NeMo models\n", + "import torch.multiprocessing as mp\n", + "mp.set_start_method(\"spawn\", force=True)\n", + "\n", "from nvflare import SimulatorRunner \n", "\n", "simulator = SimulatorRunner(\n", @@ -310,7 +316,7 @@ }, { "cell_type": "markdown", - "id": "bccf7bed", + "id": "2e56653f", "metadata": {}, "source": [ "#### 2. Federated training\n", @@ -323,7 +329,7 @@ { "cell_type": "code", "execution_count": null, - "id": "782af9c0", + "id": "ad3406a6", "metadata": { "scrolled": true }, @@ -333,7 +339,7 @@ "num_rounds=5\n", "trainer_config=\"trainer.max_steps\\=200 trainer.val_check_interval\\=100\"\n", "\n", - "!nvflare job create -force -j \"./jobs/peft_{peft_scheme}_fedavg_345M\" -w \"sag_nemo\" -sd \"code\" \\\n", + "!nvflare job create -force -j \"./jobs/peft_{peft_scheme}_fedavg_345M\" -w \"sag_nemo\" -sd \"nemo_nvflare\" \\\n", " -f app_1/config_fed_client.conf app_script={app_script} app_config=\"{peft_scheme_arg} model.restore_from_path\\={restore_from_path} {trainer_config} {val_files} {train_files_prefix}-1.jsonl\\]\" \\\n", " -f app_2/config_fed_client.conf app_script={app_script} app_config=\"{peft_scheme_arg} model.restore_from_path\\={restore_from_path} {trainer_config} {val_files} {train_files_prefix}-2.jsonl\\]\" \\\n", " -f app_3/config_fed_client.conf app_script={app_script} app_config=\"{peft_scheme_arg} model.restore_from_path\\={restore_from_path} {trainer_config} {val_files} {train_files_prefix}-3.jsonl\\]\" \\\n", @@ -342,7 +348,7 @@ }, { "cell_type": "markdown", - "id": "41088905", + "id": "5e591653", "metadata": {}, "source": [ "Next, simulate the federated training using FedAvg. " @@ -351,12 +357,16 @@ { "cell_type": "code", "execution_count": null, - "id": "00109b1e", + "id": "8559b79f", "metadata": { "scrolled": true }, "outputs": [], "source": [ + "# required by NeMo models\n", + "import torch.multiprocessing as mp\n", + "mp.set_start_method(\"spawn\", force=True)\n", + "\n", "from nvflare import SimulatorRunner \n", "\n", "simulator = SimulatorRunner(\n", @@ -371,25 +381,15 @@ }, { "cell_type": "markdown", - "id": "d3d8d656", - "metadata": {}, - "source": [ - "You can visualize the training process using TensorBoard" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "7f6755b6", + "id": "3e20ca56", "metadata": {}, - "outputs": [], "source": [ - "!tensorboard --logdir /tmp/nvflare/nemo" + "You can visualize the training process using TensorBoard by running `tensorboard --logdir /tmp/nvflare/nemo` in a new terminal." ] }, { "cell_type": "markdown", - "id": "d0c35f89", + "id": "a8e5b7c0", "metadata": {}, "source": [ "## Results\n", @@ -405,7 +405,7 @@ }, { "cell_type": "markdown", - "id": "7174a47a", + "id": "65833f4b", "metadata": {}, "source": [ "## Inference\n", @@ -417,7 +417,7 @@ { "cell_type": "code", "execution_count": null, - "id": "72d1d6e9", + "id": "dcf08bc6", "metadata": {}, "outputs": [], "source": [ @@ -432,7 +432,7 @@ }, { "cell_type": "markdown", - "id": "afe4ed67", + "id": "7b3667c0", "metadata": {}, "source": [ "First, we need to convert the best global PEFT model into a NeMo ckpt." @@ -441,7 +441,7 @@ { "cell_type": "code", "execution_count": null, - "id": "54f93b59", + "id": "3d08150a", "metadata": {}, "outputs": [], "source": [ @@ -456,7 +456,7 @@ }, { "cell_type": "markdown", - "id": "6311edbd", + "id": "2963d08e", "metadata": {}, "source": [ "Next, we will load the global model." @@ -465,7 +465,7 @@ { "cell_type": "code", "execution_count": null, - "id": "5b07ecbc", + "id": "9ffe513d", "metadata": {}, "outputs": [], "source": [ @@ -475,7 +475,7 @@ "from omegaconf import OmegaConf\n", "\n", "# Load model configuration inference of the global model\n", - "cfg = OmegaConf.load(\"code/megatron_gpt_peft_fl_eval_config.yaml\")\n", + "cfg = OmegaConf.load(\"nemo_nvflare/megatron_gpt_peft_fl_eval_config.yaml\")\n", "\n", "# Build trainer\n", "trainer = MegatronLMPPTrainerBuilder(cfg).create_trainer()\n", @@ -499,7 +499,7 @@ }, { "cell_type": "markdown", - "id": "b6b00b36", + "id": "59fa62cb", "metadata": {}, "source": [ "Run the model" @@ -508,7 +508,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c03a073d", + "id": "acd89469", "metadata": {}, "outputs": [], "source": [ @@ -535,7 +535,7 @@ }, { "cell_type": "markdown", - "id": "b9d8fd7c", + "id": "d14026fc", "metadata": {}, "source": [ "The expected output of a well-trained model looks something like this. Note, the test sentences do not include ground truth labels.\n", @@ -555,7 +555,7 @@ { "cell_type": "code", "execution_count": null, - "id": "3e7aaaa5", + "id": "db70a19a", "metadata": {}, "outputs": [], "source": [] diff --git a/integration/nemo/examples/prompt_learning/README.md b/integration/nemo/examples/prompt_learning/README.md index a19fc8744e..c25457496e 100644 --- a/integration/nemo/examples/prompt_learning/README.md +++ b/integration/nemo/examples/prompt_learning/README.md @@ -3,7 +3,7 @@ In this example, we utilize NeMo's [prompt learning](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/nemo_megatron/prompt_learning.html) feature to showcase how to adapt a large language model (LLM) to a downstream task such as financial sentiment predictions. -As the prompt learning technique shown in the example is p-tuning which adds a small prompt encoder network to the LLM +As the prompt learning technique shown in the example is p-tuning, which adds a small prompt encoder network to the LLM to produce virtual tokens that guide the model toward the desired output of the downstream task. @@ -13,14 +13,24 @@ In our federated implementation, the LLM parameters stay fixed. Prompt encoder p ## Dependencies -We assume you followed the instructions [here](../../README.md#requirements) -to install the NeMo, NVFlare, and the NeMo-NVFlare package. - The example was tested with the [NeMo 23.02 container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo). +In the following, we assume this example folder of the container is mounted to `/workspace` and all downloading, etc. operations are based on this root path. + +Start the docker container using +``` +DOCKER_IMAGE="nvcr.io/nvidia/nemo:23.02" +docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 \ +-v ${PWD}:/workspace -w /workspace ${DOCKER_IMAGE} +``` + +For easy experimentation with NeMo, install NVFlare and mount the code inside the [nemo_nvflare](./nemo_nvflare) folder. +``` +pip install nvflare~=2.4.0rc7 +export PYTHONPATH=${PYTHONPATH}:/workspace +``` ## Examples ### 1. Federated p-tuning using a 345 million parameter GPT model -This example requires a GPU with at least 16GB memory to run three clients in parallel on the same GPU. We use [JupyterLab](https://jupyterlab.readthedocs.io) for this example. To start JupyterLab, run ``` @@ -28,9 +38,14 @@ jupyter lab . ``` and open [prompt_learning.ipynb](./prompt_learning.ipynb). +#### Hardware requirement +This example requires a GPU with at least 16GB of memory to run three clients in parallel on the same GPU. + ### 2. Federated p-tuning using a 20 billion parameter GPT model -This example running a 20B GPT model requires more computational resources. -To run three clients in parallel, we require at least six GPUs with 64 GB memory or more each -(Ampere or later GPU architecture). +This example of running a 20B GPT model requires more computational resources. To run the example, follow the instructions in [prompt_learning_20B.md](prompt_learning_20B.md). + +#### Hardware requirement +To run three clients in parallel, we require at least six GPUs with 64 GB memory or more each +(Ampere or later GPU architecture). diff --git a/integration/nemo/nemo_nvflare/__init__.py b/integration/nemo/examples/prompt_learning/nemo_nvflare/__init__.py similarity index 84% rename from integration/nemo/nemo_nvflare/__init__.py rename to integration/nemo/examples/prompt_learning/nemo_nvflare/__init__.py index f109d45c1c..802119c693 100644 --- a/integration/nemo/nemo_nvflare/__init__.py +++ b/integration/nemo/examples/prompt_learning/nemo_nvflare/__init__.py @@ -13,12 +13,8 @@ # limitations under the License. from .config_sharer import ConfigSharer -from .config_sharer_sft import ConfigSharerSFT from .fed_megatron_gpt_prompt_learning_model import FedMegatronGPTPromptLearningModel from .learner_executor import NemoLearnerExecutor from .prompt_encoder import ServerPromptEncoder from .prompt_learner import PromptLearner -from .server_sft_model import ServerSFTModel -from .sft_learner import SFTLearner from .share_config import ShareConfig -from .share_config_sft import ShareConfigSFT diff --git a/integration/nemo/nemo_nvflare/config_sharer.py b/integration/nemo/examples/prompt_learning/nemo_nvflare/config_sharer.py similarity index 100% rename from integration/nemo/nemo_nvflare/config_sharer.py rename to integration/nemo/examples/prompt_learning/nemo_nvflare/config_sharer.py diff --git a/integration/nemo/nemo_nvflare/constants.py b/integration/nemo/examples/prompt_learning/nemo_nvflare/constants.py similarity index 100% rename from integration/nemo/nemo_nvflare/constants.py rename to integration/nemo/examples/prompt_learning/nemo_nvflare/constants.py diff --git a/integration/nemo/nemo_nvflare/fed_megatron_gpt_prompt_learning_model.py b/integration/nemo/examples/prompt_learning/nemo_nvflare/fed_megatron_gpt_prompt_learning_model.py similarity index 100% rename from integration/nemo/nemo_nvflare/fed_megatron_gpt_prompt_learning_model.py rename to integration/nemo/examples/prompt_learning/nemo_nvflare/fed_megatron_gpt_prompt_learning_model.py diff --git a/integration/nemo/nemo_nvflare/learner_executor.py b/integration/nemo/examples/prompt_learning/nemo_nvflare/learner_executor.py similarity index 100% rename from integration/nemo/nemo_nvflare/learner_executor.py rename to integration/nemo/examples/prompt_learning/nemo_nvflare/learner_executor.py diff --git a/integration/nemo/nemo_nvflare/prompt_encoder.py b/integration/nemo/examples/prompt_learning/nemo_nvflare/prompt_encoder.py similarity index 100% rename from integration/nemo/nemo_nvflare/prompt_encoder.py rename to integration/nemo/examples/prompt_learning/nemo_nvflare/prompt_encoder.py diff --git a/integration/nemo/nemo_nvflare/prompt_learner.py b/integration/nemo/examples/prompt_learning/nemo_nvflare/prompt_learner.py similarity index 100% rename from integration/nemo/nemo_nvflare/prompt_learner.py rename to integration/nemo/examples/prompt_learning/nemo_nvflare/prompt_learner.py diff --git a/integration/nemo/nemo_nvflare/share_config.py b/integration/nemo/examples/prompt_learning/nemo_nvflare/share_config.py similarity index 100% rename from integration/nemo/nemo_nvflare/share_config.py rename to integration/nemo/examples/prompt_learning/nemo_nvflare/share_config.py diff --git a/integration/nemo/nemo_nvflare/utils.py b/integration/nemo/examples/prompt_learning/nemo_nvflare/utils.py similarity index 100% rename from integration/nemo/nemo_nvflare/utils.py rename to integration/nemo/examples/prompt_learning/nemo_nvflare/utils.py diff --git a/integration/nemo/examples/prompt_learning/prompt_learning.ipynb b/integration/nemo/examples/prompt_learning/prompt_learning.ipynb index ef51377001..4d76fcd860 100644 --- a/integration/nemo/examples/prompt_learning/prompt_learning.ipynb +++ b/integration/nemo/examples/prompt_learning/prompt_learning.ipynb @@ -2,7 +2,7 @@ "cells": [ { "cell_type": "markdown", - "id": "b43584a8", + "id": "56e442d4", "metadata": {}, "source": [ "# Prompt Learning with NeMo\n", @@ -19,17 +19,17 @@ }, { "cell_type": "markdown", - "id": "578585e4", + "id": "6dac11e2", "metadata": {}, "source": [ "## Dependencies\n", - "We assume you followed the instructions [here](../../README.md#requirements) \n", - "to install the NeMo framework and the NeMo-NVFlare package. " + "We assume you followed the instructions [here](./README.md) \n", + "to install the NeMo and NVFlare frameworks and mount the required codes." ] }, { "cell_type": "markdown", - "id": "199f1fe5", + "id": "47b1d4dc", "metadata": {}, "source": [ "## Download the pre-trained LLM\n", @@ -39,7 +39,7 @@ { "cell_type": "code", "execution_count": null, - "id": "4ac31bab", + "id": "581035ee", "metadata": {}, "outputs": [], "source": [ @@ -51,7 +51,7 @@ { "cell_type": "code", "execution_count": null, - "id": "9a14ccb9", + "id": "154be2b0", "metadata": {}, "outputs": [], "source": [ @@ -66,7 +66,7 @@ }, { "cell_type": "markdown", - "id": "9b4b0a65", + "id": "9a420d6e", "metadata": {}, "source": [ "## Data preprocessing\n", @@ -77,7 +77,7 @@ }, { "cell_type": "markdown", - "id": "f4a845d4", + "id": "3b3d4155", "metadata": {}, "source": [ "#### 1. Download the preprocessing scripts\n", @@ -87,7 +87,7 @@ { "cell_type": "code", "execution_count": null, - "id": "3a00456f", + "id": "f5c33254", "metadata": {}, "outputs": [], "source": [ @@ -100,19 +100,31 @@ }, { "cell_type": "markdown", - "id": "353a28e0", + "id": "974248b8", "metadata": {}, "source": [ "#### 2. Download the Financial PhraseBank Dataset\n", "\n", "Download the `FinancialPhraseBank-v1.0.zip` dataset from [here](https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v1.0/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v1.0.zip).\n", "\n", - "Then extract it under `./data`." + "Then extract it under `./data`. Note, after extraction, the data folder should have the following content\n", + "```\n", + "data\n", + "├── FinancialPhraseBank-v1.0\n", + "│   ├── License.txt\n", + "│   ├── README.txt\n", + "│   ├── Sentences_50Agree.txt\n", + "│   ├── Sentences_66Agree.txt\n", + "│   ├── Sentences_75Agree.txt\n", + "│   └── Sentences_AllAgree.txt\n", + "├── FinancialPhraseBank-v1.0.zip\n", + "└── split_financial_phrase_data.py\n", + "```" ] }, { "cell_type": "markdown", - "id": "12bb6682", + "id": "b1f8ad50", "metadata": {}, "source": [ "#### 3. Preprocess the dataset" @@ -121,7 +133,7 @@ { "cell_type": "code", "execution_count": null, - "id": "2ceb4180", + "id": "fbbf86af", "metadata": {}, "outputs": [], "source": [ @@ -130,7 +142,7 @@ }, { "cell_type": "markdown", - "id": "baa61a74", + "id": "29aaffee", "metadata": {}, "source": [ "#### 4. Split the dataset to simulate clients\n", @@ -140,7 +152,7 @@ { "cell_type": "code", "execution_count": null, - "id": "339884a1", + "id": "725115cc", "metadata": {}, "outputs": [], "source": [ @@ -149,7 +161,7 @@ }, { "cell_type": "markdown", - "id": "cbcab01b", + "id": "7a45c985", "metadata": {}, "source": [ "## Federated learning simulations\n", @@ -161,7 +173,7 @@ }, { "cell_type": "markdown", - "id": "4fbc7c4c", + "id": "8b56fc06", "metadata": {}, "source": [ "#### 1. Local P-Tuning\n", @@ -172,7 +184,7 @@ { "cell_type": "code", "execution_count": null, - "id": "14896baa", + "id": "7dd7e496", "metadata": {}, "outputs": [], "source": [ @@ -181,7 +193,7 @@ }, { "cell_type": "markdown", - "id": "1ea16b74", + "id": "1529d090", "metadata": {}, "source": [ "Next, simulate each client p-tuning on their local dataset using the FL simulator. To do this, we only run 1 round of FL, with each client running 50 p-tuning epochs on their local dataset." @@ -190,7 +202,7 @@ { "cell_type": "code", "execution_count": null, - "id": "5abf5055", + "id": "6a05bdd8", "metadata": { "scrolled": true }, @@ -210,7 +222,7 @@ }, { "cell_type": "markdown", - "id": "f0bb49cb", + "id": "6456327c", "metadata": {}, "source": [ "#### 2. Federated P-Tuning\n", @@ -221,7 +233,7 @@ { "cell_type": "code", "execution_count": null, - "id": "5151467a", + "id": "c6ec1399", "metadata": {}, "outputs": [], "source": [ @@ -230,7 +242,7 @@ }, { "cell_type": "markdown", - "id": "eadb0a5c", + "id": "e5083061", "metadata": {}, "source": [ "Next, simulate the federated p-tuning using FedAvg. Here, each client p-tunes for one local epoch before sending their local model updates to the server for aggregation. This is repeated for 50 FL rounds." @@ -239,7 +251,7 @@ { "cell_type": "code", "execution_count": null, - "id": "eea2c83a", + "id": "38609850", "metadata": { "scrolled": true }, @@ -259,25 +271,15 @@ }, { "cell_type": "markdown", - "id": "a9276ce2", + "id": "9fc069d3", "metadata": {}, "source": [ - "You can visualize the training process using TensorBoard" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "5c93483c", - "metadata": {}, - "outputs": [], - "source": [ - "!tensorboard --logdir /tmp/nvflare/nemo" + "You can visualize the training process using TensorBoard by running `tensorboard --logdir /tmp/nvflare/nemo` in a new terminal." ] }, { "cell_type": "markdown", - "id": "0e6763ca", + "id": "aad25a10", "metadata": {}, "source": [ "## Results\n", @@ -288,7 +290,7 @@ }, { "cell_type": "markdown", - "id": "639e95aa", + "id": "fbbac75c", "metadata": {}, "source": [ "## Inference\n", @@ -300,7 +302,7 @@ { "cell_type": "code", "execution_count": null, - "id": "38ae679d", + "id": "52bb91c3", "metadata": {}, "outputs": [], "source": [ @@ -315,7 +317,7 @@ }, { "cell_type": "markdown", - "id": "23ce4e16", + "id": "e5740fbf", "metadata": {}, "source": [ "Next, we will load the global model." @@ -324,7 +326,7 @@ { "cell_type": "code", "execution_count": null, - "id": "c02f046e", + "id": "605d0d1c", "metadata": {}, "outputs": [], "source": [ @@ -369,7 +371,7 @@ }, { "cell_type": "markdown", - "id": "253cdc30", + "id": "7bf97036", "metadata": {}, "source": [ "Overwrite the prompt encoder with the best global model" @@ -378,7 +380,7 @@ { "cell_type": "code", "execution_count": null, - "id": "0f257854", + "id": "33f9771b", "metadata": {}, "outputs": [], "source": [ @@ -391,7 +393,7 @@ }, { "cell_type": "markdown", - "id": "57b954e7", + "id": "69c35011", "metadata": {}, "source": [ "Run the model" @@ -400,7 +402,7 @@ { "cell_type": "code", "execution_count": null, - "id": "8781d8f2", + "id": "64402b65", "metadata": {}, "outputs": [], "source": [ @@ -414,7 +416,7 @@ }, { "cell_type": "markdown", - "id": "12613bbd", + "id": "f1ff31f1", "metadata": {}, "source": [ "The expected output predictions look something like this\n", @@ -434,7 +436,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d69d4973", + "id": "98d73f49", "metadata": {}, "outputs": [], "source": [] @@ -456,7 +458,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.6" + "version": "3.10.12" } }, "nbformat": 4, diff --git a/integration/nemo/examples/prompt_learning/prompt_learning_20B.md b/integration/nemo/examples/prompt_learning/prompt_learning_20B.md index 97bb03a2d2..1e2d54ba48 100644 --- a/integration/nemo/examples/prompt_learning/prompt_learning_20B.md +++ b/integration/nemo/examples/prompt_learning/prompt_learning_20B.md @@ -15,13 +15,18 @@ To run three clients in parallel, we require at least six GPUs with 64 GB memory (Ampere or later GPU architecture). The example was tested on 6xA100 GPUs with 80 GB each. -We assume you followed the instructions [here](../../README.md#requirements) -to install the NeMo framework and the NeMo-NVFlare package. +We assume you followed the instructions [here](./README.md) +to install the NeMo framework and mount the required code. The example was tested using the [NeMo Docker container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo), available with `docker pull nvcr.io/nvidia/nemo:23.02`. For downloading the pre-trained model, we use [git lfs](https://git-lfs.com). +Install it in the container with +``` +apt update +apt install git-lfs +``` ## Download the pre-trained LLM In this example, we use a [Megatron-GPT 20B](https://huggingface.co/nvidia/nemo-megatron-gpt-20B), a transformer-based language model based on the GPT architecture. @@ -29,7 +34,8 @@ In this example, we use a [Megatron-GPT 20B](https://huggingface.co/nvidia/nemo- # download the model from HuggingFace using git lfs git clone https://huggingface.co/nvidia/nemo-megatron-gpt-20B ``` -After download, the checkpoint `nemo_gpt20B_bf16_tp4.nemo` should have a size of 38 GB. +> Note, this will take some time. After download, the checkpoint `nemo_gpt20B_bf16_tp4.nemo` should have a size of 38 GB. +> You can check the download status with `du -sh nemo-megatron-gpt-20B/nemo_gpt20B_bf16_tp4.nemo`. Next, in order to minimize the number of GPUs needed to simulate each client, we convert the downloaded checkpoint that was trained using tensor parallel of size 4, to tensor parallel of size 2. @@ -115,27 +121,30 @@ In a standard terminal, run ``` python3 create_configs.py --job_folder "jobs/gpt_p-tuning_local_20B" --num_clients 3 --devices 2 --aggregation_epochs 50 --num_rounds 1 ``` -Next, submit the federated p-tuning job using the admin prompt. -Replace `[PWD]` with the path to this directory. +Next, submit the federated p-tuning job in the terminal running the admin command prompt. + ``` -submit_job [PWD]/jobs/gpt_p-tuning_local_20B +submit_job /workspace/jobs/gpt_p-tuning_local_20B ``` #### 2. Federated P-Tuning We use the [FedAvg](https://arxiv.org/abs/1602.05629) algorithm to p-tune the model in a federated scenario. First, create and modify the configuration files again. This time, we increase the number of FL rounds and decrease the number of local epochs per round to match the federated scenario. -Here, each client p-tunes for one local epoch before sending their local model updates to the server for aggregation. This is repeated for 50 FL rounds. +Here, each client p-tunes for one local epoch before sending their local model updates to the server for aggregation. +This is repeated for 50 FL rounds. + +In a standard terminal, run ``` python3 create_configs.py --job_folder "jobs/gpt_p-tuning_fedavg_20B" --num_clients 3 --devices 2 --aggregation_epochs 1 --num_rounds 50 ``` -Next, simulate the federated p-tuning using FedAvg. +Next, simulate the federated p-tuning using FedAvg in the terminal running the admin command prompt. ``` -submit_job [PWD]/jobs/gpt_p-tuning_fedavg_20B +submit_job /workspace/jobs/gpt_p-tuning_fedavg_20B ``` You can visualize the training process using TensorBoard ``` -tensorboard --logdir /tmp/nvflare/nemo +tensorboard --logdir /tmp/nvflare/poc ``` ## Results diff --git a/integration/nemo/examples/supervised_fine_tuning/README.md b/integration/nemo/examples/supervised_fine_tuning/README.md index 9fbce4cc3c..610814d3d8 100644 --- a/integration/nemo/examples/supervised_fine_tuning/README.md +++ b/integration/nemo/examples/supervised_fine_tuning/README.md @@ -1,22 +1,39 @@ ## Supervised Fine-tuning (SFT) with NeMo In this example, we utilize NeMo's [supervised fine-tuning](https://github.com/NVIDIA/NeMo-Megatron-Launcher#515-instruction-following-via-supervised-finetuning--sft-) -feature to showcase how to fine-tune the whole model on supervised data for learning how to follow user specified instructions. +feature to showcase how to fine-tune the whole model on supervised data for learning how to follow user-specified instructions. Due to the large model size of the LLM, we use NVFlare's streaming feature to transfer the model in chunks. -## Dependencies -This example running a 1.3B GPT model requires considerable computational resources. For training 1.3B model, SFT needs ~24GB GPU memory using fp16 precision. Hence, to run three clients in parallel, we can compute the resource needed accordingly. - +## Hardware requirement The example for a 3-client 1.3B GPT model experiment can be performed on either three 32 GB V100 GPUs, or one 80 GB A100 GPU. -We assume you followed the instructions [here](../../README.md#requirements) -to install the NeMo, NVFlare, and the NeMo-NVFlare package. +## Dependencies +This example of running a 1.3B GPT model requires considerable computational resources. For training 1.3B model, SFT needs ~24GB GPU memory using fp16 precision. Hence, we can compute the resources needed accordingly to run three clients in parallel. The example was tested using the [NeMo Docker container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo), -available with `docker pull nvcr.io/nvidia/nemo:23.02`. In the following, we assume the root folder of the container is mounted to `/workspace` and all downloading, etc. operations are based on this root path. +available with `docker pull nvcr.io/nvidia/nemo:23.06`. +In the following, we assume this example folder of the container is mounted to `/workspace` and all downloading, etc. operations are based on this root path. -For downloading the pre-trained model, we use [git lfs](https://git-lfs.com). +Start the docker container using +``` +DOCKER_IMAGE="nvcr.io/nvidia/nemo:23.06" +docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 \ +-v ${PWD}:/workspace -w /workspace ${DOCKER_IMAGE} +``` + +For easy experimentation with NeMo, install NVFlare and mount the code inside the [nemo_nvflare](./nemo_nvflare) folder. +``` +pip install nvflare~=2.4.0rc7 +export PYTHONPATH=${PYTHONPATH}:/workspace +``` + +To download the pre-trained model, we use [git lfs](https://git-lfs.com). +Install it in the container with +``` +apt update +apt install git-lfs +``` ## Download the pre-trained LLM In this example, we use [Megatron-GPT 1.3B](https://huggingface.co/nvidia/nemo-megatron-gpt-1.3B), a transformer-based language model based on the GPT architecture. @@ -34,9 +51,9 @@ For SFT task, we will use three datasets: - [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) - [OpenAssistant Conversations](https://huggingface.co/datasets/OpenAssistant/oasst1) -These three datasets contain instruction-following data in different formats under different settings: oasst1 features a tree struture for full conversations, while the other two are instruction(w/ or w/o context)-response pairs. +These three datasets contain instruction-following data in different formats under different settings: oasst1 features a tree structure for full conversations, while the other two are instruction(w/ or w/o context)-response pairs. -In this example, we first preprocess them following the [NeMo SFT](https://github.com/NVIDIA/NeMo-Megatron-Launcher#5151-sft-data-formatting)'s instructions. The script converts the "Instruction", "Context" and "Response" fields (or their equivalents) into "Input" and "Output". The script also concatenates the "Instruction" and "Context" fields with a \n\n separator and randomizes the order in which they appear in the input to generate a new JSONL file. +In this example, we first preprocess them following the [NeMo SFT](https://github.com/NVIDIA/NeMo-Megatron-Launcher#5151-sft-data-formatting) instructions. The script converts the "Instruction", "Context" and "Response" fields (or their equivalents) into "Input" and "Output". The script also concatenates the "Instruction" and "Context" fields with a \n\n separator and randomizes the order in which they appear in the input to generate a new JSONL file. #### 1. Download the datasets We download the datasets from HuggingFace: @@ -62,7 +79,7 @@ python utils/preprocess_oasst1.py --training_file Data/oasst1/data/train-00000-o ``` #### 3. Combine for centralized training -We also generate a combined version for centralized training baseline: +We also generate a combined version for a centralized training baseline: ``` mkdir Data/Processed/combined python utils/combine_jsonl.py --file_list Data/Processed/alpaca/training.jsonl Data/Processed/dolly/training.jsonl Data/Processed/oasst1/training.jsonl --output_path Data/Processed/combined/training.jsonl @@ -110,7 +127,7 @@ nvflare simulator jobs/gpt_sft_1.3B_fedavg -w workspace_simulator_fedavg -n 3 -g ``` ### Use POC mode -Alternatively, we can also NVFlare's [POC mode](https://nvflare.readthedocs.io/en/main/getting_started.html#setting-up-poc) to simulate +Alternatively, we can also use NVFlare's [POC mode](https://nvflare.readthedocs.io/en/main/getting_started.html#setting-up-poc) to simulate #### 1. Local and Centralized SFT For single-site and centralized training experiments, we create the poc workspaces: @@ -127,7 +144,7 @@ nvflare poc start -p admin@nvidia.com ``` -Next, copy the jobs to temp workspace. +Next, copy the jobs to the temp workspace. ``` cp -r jobs/gpt_sft_1.3B_* /tmp/nvflare/poc/example_project/prod_00/admin\@nvidia.com/transfer/ ``` @@ -139,6 +156,11 @@ submit_job gpt_sft_1.3B_dolly submit_job gpt_sft_1.3B_oasst1 submit_job gpt_sft_1.3B_combined ``` +During training, we can visualize the training process using TensorBoard. +With FL simulator, use +``` +tensorboard --logdir /workspace +``` #### 2. Federated SFT We use the [FedAvg](https://arxiv.org/abs/1602.05629) algorithm to perform SFT on the model in a federated scenario with 3 clients, each uses one of the three datasets. @@ -157,7 +179,7 @@ nvflare poc start -p admin@nvidia.com ``` -Next, simulate the federated SFT using FedAvg, similarly to single-client experiments +Next, simulate the federated SFT using FedAvg, similarly to single-client experiments: ``` cp -r jobs/gpt_sft_1.3B_fedavg /tmp/nvflare/poc/example_project/prod_00/admin\@nvidia.com/transfer/ ``` @@ -166,11 +188,14 @@ and to submit the FedAvg job submit_job gpt_sft_1.3B_fedavg ``` -## Results -During training, we can visualize the training process using TensorBoard +During training, we can visualize the training process using TensorBoard. +With the POC mode, use ``` -tensorboard --logdir /tmp/nvflare/nemo +tensorboard --logdir /tmp/nvflare/poc ``` + +## Results + In this scenario, all experiments utilize the same validation set, allowing for a direct comparison across all models. Note that we ran FL for 5 rounds, and asked NeMo to record the validation losses every few steps during local training. The validation losses for all experiments are shown below. @@ -203,7 +228,7 @@ As shown, FedAvg is able to generate a model with the best overall performance. We use NeMo's [inference script](https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/language_modeling/megatron_gpt_eval.py) for generation task with models after SFT. Below, we define some test examples to feed to the SFT model to see its predictions. -First, we ask the model to generate answer to an open question "Tell me an interesting fact about space travel." +First, we ask the model to generate an answer to an open question: "Tell me an interesting fact about space travel." ``` ALPACA: The first human to orbit the Earth was Neil Armstrong, who flew on the Apollo 11 mission in 1969.' DOLLY: The International Space Station is the largest floating structure in the universe. It is made of steel and is about the size of a small house. @@ -211,7 +236,7 @@ OASST: Sure! Here are a few interesting facts about space travel:\n\n1. Space tr COMBINED: The first human to set foot on the Moon was Neil Armstrong. FEDAVG: The first person to travel to space was Neil Armstrong, who set foot on the moon in 1969. ``` -Note that models mostly gives plausible answers, but ALPACA-finetuned model in fact gives misinformation, since it should be Yuri Gagarin who is the first human to orbit the Earth. +Note that models mostly give plausible answers, but the ALPACA-finetuned model, in fact, gives misinformation since it should be Yuri Gagarin who is the first human to orbit the Earth. On the other hand, the model trained on the combined dataset, as well as the FL model trained with FedAvg, are able to generate a more accurate answer. Next, we ask the model to answer a question according to a given context, one instance from [SQuAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). @@ -228,6 +253,6 @@ OASST: The Denver Broncos defeated the Carolina Panthers 24–10 to win the Supe COMBINED: The Denver Broncos' FEDAVG: The AFC champion Denver Broncos defeated the NFC champion Carolina Panthers 24–10 to win the Super Bowl.' ``` -As we can see, the key word "Denver Broncos" is correctly captured by all models. However, ALPACA and FedAvg answers are a bit redundant, and OASST answer is not directly "to the question". +As we can see, the keyword "Denver Broncos" is correctly captured by all models. However, ALPACA and FedAvg answers are a bit redundant, and OASST answer is not directly "to the question". -Based on the above results, we can see that the models trained on the combined dataset and in a federated fashion are able to generate more stable and accurate answers. +Based on the above results, we can see that the models trained on the combined dataset and in a federated fashion can generate more stable and accurate answers. diff --git a/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/__init__.py b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/__init__.py new file mode 100644 index 0000000000..18e75a481b --- /dev/null +++ b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/__init__.py @@ -0,0 +1,19 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .config_sharer_sft import ConfigSharerSFT +from .learner_executor import NemoLearnerExecutor +from .server_sft_model import ServerSFTModel +from .sft_learner import SFTLearner +from .share_config_sft import ShareConfigSFT diff --git a/integration/nemo/nemo_nvflare/config_sharer_sft.py b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/config_sharer_sft.py similarity index 100% rename from integration/nemo/nemo_nvflare/config_sharer_sft.py rename to integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/config_sharer_sft.py diff --git a/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/constants.py b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/constants.py new file mode 100644 index 0000000000..2c54b42039 --- /dev/null +++ b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/constants.py @@ -0,0 +1,23 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +class NemoConstants(object): + TASK_SHARE_CONFIG = "share_config" + + +class NemoDataKind(object): + CONFIGS = "nemo_configs" + NEMO_CONFIG = "nemo_config" + TASK_TEMPLATES = "nemo_task_templates" diff --git a/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/learner_executor.py b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/learner_executor.py new file mode 100644 index 0000000000..a8fccbb9a5 --- /dev/null +++ b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/learner_executor.py @@ -0,0 +1,80 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from nvflare.apis.dxo import from_shareable +from nvflare.apis.fl_constant import ReturnCode +from nvflare.apis.fl_context import FLContext +from nvflare.apis.shareable import Shareable, make_reply +from nvflare.apis.signal import Signal +from nvflare.app_common.app_constant import AppConstants +from nvflare.app_common.executors.learner_executor import LearnerExecutor + +from .constants import NemoConstants, NemoDataKind + + +class NemoLearnerExecutor(LearnerExecutor): + def __init__( + self, + learner_id, + train_task=AppConstants.TASK_TRAIN, + submit_model_task=AppConstants.TASK_SUBMIT_MODEL, + validate_task=AppConstants.TASK_VALIDATION, + share_config_task=NemoConstants.TASK_SHARE_CONFIG, + ): + """Key component to run learner on clients. + + Args: + learner_id (str): id of the learner object + train_task (str, optional): task name for train. Defaults to AppConstants.TASK_TRAIN. + submit_model_task (str, optional): task name for submit model. Defaults to AppConstants.TASK_SUBMIT_MODEL. + validate_task (str, optional): task name for validation. Defaults to AppConstants.TASK_VALIDATION. + share_config_task (str, optional): share config task name. + """ + super().__init__( + learner_id=learner_id, + train_task=train_task, + submit_model_task=submit_model_task, + validate_task=validate_task, + ) + self.share_config_task = share_config_task + self.is_initialized = False + + def execute(self, task_name: str, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal) -> Shareable: + if not self.is_initialized: + self.is_initialized = True + self.initialize(fl_ctx) + + if task_name == self.share_config_task: + self.log_info(fl_ctx, f"Client trainer got task: {task_name}") + try: + return self._set_learner_configs(shareable, fl_ctx, abort_signal) + except Exception as e: + self.log_error(fl_ctx, f"Setting config failed with exception {e}") + return make_reply(ReturnCode.EXECUTION_EXCEPTION) + else: + return super().execute(task_name=task_name, shareable=shareable, fl_ctx=fl_ctx, abort_signal=abort_signal) + + def _set_learner_configs(self, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal) -> Shareable: + dxo = from_shareable(shareable) + + if dxo.data_kind != NemoDataKind.CONFIGS: + raise ValueError(f"Expected DXO data to be of kind NemoDataKind.CONFIGS but got {dxo.data_kind}") + + if not dxo.data: + raise ValueError("Received config data is empty!") + + self.learner.set_configs(configs=dxo.data) + self.log_info(fl_ctx, f"Received config with {len(dxo.data)} entries from server.") + + return make_reply(ReturnCode.OK) diff --git a/integration/nemo/nemo_nvflare/server_sft_model.py b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/server_sft_model.py similarity index 100% rename from integration/nemo/nemo_nvflare/server_sft_model.py rename to integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/server_sft_model.py diff --git a/integration/nemo/nemo_nvflare/sft_learner.py b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/sft_learner.py similarity index 100% rename from integration/nemo/nemo_nvflare/sft_learner.py rename to integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/sft_learner.py diff --git a/integration/nemo/nemo_nvflare/share_config_sft.py b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/share_config_sft.py similarity index 100% rename from integration/nemo/nemo_nvflare/share_config_sft.py rename to integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/share_config_sft.py diff --git a/integration/nemo/nemo_nvflare/utils_sft.py b/integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/utils_sft.py similarity index 100% rename from integration/nemo/nemo_nvflare/utils_sft.py rename to integration/nemo/examples/supervised_fine_tuning/nemo_nvflare/utils_sft.py From 0b4bac85a44aa41399e5f3ba8dcd9a9e76d65665 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Mon, 29 Jan 2024 21:08:30 -0800 Subject: [PATCH 21/39] Update all examples/research/integration requirements (#2330) --- examples/advanced/brats18/requirements.txt | 2 +- examples/advanced/cifar10/cifar10-real-world/requirements.txt | 2 +- examples/advanced/cifar10/cifar10-sim/requirements.txt | 2 +- examples/advanced/custom_authentication/requirements.txt | 2 +- examples/advanced/experiment-tracking/mlflow/requirements.txt | 2 +- .../advanced/experiment-tracking/tensorboard/requirements.txt | 4 +++- examples/advanced/experiment-tracking/wandb/requirements.txt | 4 +++- examples/advanced/federated-policies/requirements.txt | 2 +- .../advanced/federated-statistics/df_stats/requirements.txt | 2 +- .../federated-statistics/image_stats/requirements.txt | 2 +- examples/advanced/finance/requirements.txt | 2 +- examples/advanced/job-level-authorization/requirements.txt | 2 +- .../advanced/keycloak-site-authentication/requirements.txt | 2 +- examples/advanced/llm_hf/requirements.txt | 2 +- examples/advanced/nlp-ner/requirements.txt | 2 +- examples/advanced/prostate/requirements.txt | 2 +- examples/advanced/psi/user_email_match/requirements.txt | 2 +- examples/advanced/random_forest/requirements.txt | 2 +- examples/advanced/sklearn-kmeans/requirements.txt | 2 +- examples/advanced/sklearn-linear/requirements.txt | 2 +- examples/advanced/sklearn-svm/requirements.txt | 2 +- examples/advanced/swarm_learning/requirements.txt | 2 +- .../cifar10-splitnn/requirements.txt | 2 +- examples/advanced/vertical_xgboost/requirements.txt | 2 +- examples/advanced/xgboost/histogram-based/requirements.txt | 2 +- examples/advanced/xgboost/tree-based/requirements.txt | 2 +- examples/hello-world/hello-ccwf/requirements.txt | 1 + examples/hello-world/hello-cyclic/requirements.txt | 2 +- examples/hello-world/hello-numpy-cross-val/requirements.txt | 2 +- examples/hello-world/hello-numpy-sag/requirements.txt | 2 +- examples/hello-world/hello-pt/requirements.txt | 4 +++- examples/hello-world/hello-tf2/requirements.txt | 2 +- examples/hello-world/ml-to-fl/np/requirements.txt | 1 + examples/hello-world/ml-to-fl/pt/requirements.txt | 4 +++- examples/hello-world/ml-to-fl/tf/requirements.txt | 2 +- .../hello-world/step-by-step/cifar10/stats/requirements.txt | 2 +- examples/hello-world/step-by-step/requirements.txt | 2 +- .../examples/spleen_ct_segmentation_local/requirements.txt | 2 +- .../examples/spleen_ct_segmentation_sim/requirements.txt | 2 +- integration/nemo/examples/prompt_learning/requirements.txt | 2 +- integration/sample/README.md | 2 +- research/auto-fed-rl/requirements.txt | 2 +- research/fed-ce/requirements.txt | 2 +- research/fed-sm/requirements.txt | 2 +- research/one-shot-vfl/requirements.txt | 2 +- research/quantifying-data-leakage/requirements.txt | 2 +- 46 files changed, 54 insertions(+), 44 deletions(-) create mode 100644 examples/hello-world/hello-ccwf/requirements.txt create mode 100644 examples/hello-world/ml-to-fl/np/requirements.txt diff --git a/examples/advanced/brats18/requirements.txt b/examples/advanced/brats18/requirements.txt index b7dd1625cf..5757f5b0ea 100644 --- a/examples/advanced/brats18/requirements.txt +++ b/examples/advanced/brats18/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/examples/advanced/cifar10/cifar10-real-world/requirements.txt b/examples/advanced/cifar10/cifar10-real-world/requirements.txt index 9b6376874b..2fd21968dc 100644 --- a/examples/advanced/cifar10/cifar10-real-world/requirements.txt +++ b/examples/advanced/cifar10/cifar10-real-world/requirements.txt @@ -1,4 +1,4 @@ -nvflare[HE]>=2.3.0 +nvflare[HE]~=2.4.0rc torch torchvision tensorboard diff --git a/examples/advanced/cifar10/cifar10-sim/requirements.txt b/examples/advanced/cifar10/cifar10-sim/requirements.txt index 0804527963..3bbfea441b 100644 --- a/examples/advanced/cifar10/cifar10-sim/requirements.txt +++ b/examples/advanced/cifar10/cifar10-sim/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/examples/advanced/custom_authentication/requirements.txt b/examples/advanced/custom_authentication/requirements.txt index d556c8d097..e4605852b5 100644 --- a/examples/advanced/custom_authentication/requirements.txt +++ b/examples/advanced/custom_authentication/requirements.txt @@ -1 +1 @@ -nvflare>=2.4.0 +nvflare~=2.4.0rc diff --git a/examples/advanced/experiment-tracking/mlflow/requirements.txt b/examples/advanced/experiment-tracking/mlflow/requirements.txt index 811b9aa432..04a1d06c48 100644 --- a/examples/advanced/experiment-tracking/mlflow/requirements.txt +++ b/examples/advanced/experiment-tracking/mlflow/requirements.txt @@ -1,4 +1,4 @@ -nvflare[PT]>=2.3.0 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/examples/advanced/experiment-tracking/tensorboard/requirements.txt b/examples/advanced/experiment-tracking/tensorboard/requirements.txt index d96a108e51..3bbfea441b 100644 --- a/examples/advanced/experiment-tracking/tensorboard/requirements.txt +++ b/examples/advanced/experiment-tracking/tensorboard/requirements.txt @@ -1,2 +1,4 @@ -nvflare[PT]>=2.3.0 +nvflare~=2.4.0rc +torch +torchvision tensorboard diff --git a/examples/advanced/experiment-tracking/wandb/requirements.txt b/examples/advanced/experiment-tracking/wandb/requirements.txt index 7ea490208f..ad3f6241d2 100644 --- a/examples/advanced/experiment-tracking/wandb/requirements.txt +++ b/examples/advanced/experiment-tracking/wandb/requirements.txt @@ -1,3 +1,5 @@ -nvflare[PT]>=2.3.0 +nvflare~=2.4.0rc +torch +torchvision tensorboard wandb diff --git a/examples/advanced/federated-policies/requirements.txt b/examples/advanced/federated-policies/requirements.txt index 3fdbf10587..e4605852b5 100644 --- a/examples/advanced/federated-policies/requirements.txt +++ b/examples/advanced/federated-policies/requirements.txt @@ -1 +1 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc diff --git a/examples/advanced/federated-statistics/df_stats/requirements.txt b/examples/advanced/federated-statistics/df_stats/requirements.txt index dc5d8c6eaf..f897a1484a 100644 --- a/examples/advanced/federated-statistics/df_stats/requirements.txt +++ b/examples/advanced/federated-statistics/df_stats/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc numpy pandas matplotlib diff --git a/examples/advanced/federated-statistics/image_stats/requirements.txt b/examples/advanced/federated-statistics/image_stats/requirements.txt index 45e20cc1ee..9e0a46f617 100644 --- a/examples/advanced/federated-statistics/image_stats/requirements.txt +++ b/examples/advanced/federated-statistics/image_stats/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc numpy monai[itk] pandas diff --git a/examples/advanced/finance/requirements.txt b/examples/advanced/finance/requirements.txt index 5348abcbce..f8a60dc996 100644 --- a/examples/advanced/finance/requirements.txt +++ b/examples/advanced/finance/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc openmined.psi==1.1.1 pandas xgboost>=1.7.0 diff --git a/examples/advanced/job-level-authorization/requirements.txt b/examples/advanced/job-level-authorization/requirements.txt index d556c8d097..e4605852b5 100644 --- a/examples/advanced/job-level-authorization/requirements.txt +++ b/examples/advanced/job-level-authorization/requirements.txt @@ -1 +1 @@ -nvflare>=2.4.0 +nvflare~=2.4.0rc diff --git a/examples/advanced/keycloak-site-authentication/requirements.txt b/examples/advanced/keycloak-site-authentication/requirements.txt index d556c8d097..e4605852b5 100644 --- a/examples/advanced/keycloak-site-authentication/requirements.txt +++ b/examples/advanced/keycloak-site-authentication/requirements.txt @@ -1 +1 @@ -nvflare>=2.4.0 +nvflare~=2.4.0rc diff --git a/examples/advanced/llm_hf/requirements.txt b/examples/advanced/llm_hf/requirements.txt index 9651b08324..171942c7af 100644 --- a/examples/advanced/llm_hf/requirements.txt +++ b/examples/advanced/llm_hf/requirements.txt @@ -1,4 +1,4 @@ -nvflare +nvflare~=2.4.0rc torch datasets tensorboard diff --git a/examples/advanced/nlp-ner/requirements.txt b/examples/advanced/nlp-ner/requirements.txt index 0bcaa6d626..678ba0bcd7 100644 --- a/examples/advanced/nlp-ner/requirements.txt +++ b/examples/advanced/nlp-ner/requirements.txt @@ -1,4 +1,4 @@ -nvflare +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/examples/advanced/prostate/requirements.txt b/examples/advanced/prostate/requirements.txt index b7dd1625cf..5757f5b0ea 100644 --- a/examples/advanced/prostate/requirements.txt +++ b/examples/advanced/prostate/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/examples/advanced/psi/user_email_match/requirements.txt b/examples/advanced/psi/user_email_match/requirements.txt index 8c42c63f15..23c6c47d5d 100644 --- a/examples/advanced/psi/user_email_match/requirements.txt +++ b/examples/advanced/psi/user_email_match/requirements.txt @@ -1,3 +1,3 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc openmined.psi==1.1.1 pandas diff --git a/examples/advanced/random_forest/requirements.txt b/examples/advanced/random_forest/requirements.txt index 02353a0589..96c88f1ec4 100644 --- a/examples/advanced/random_forest/requirements.txt +++ b/examples/advanced/random_forest/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc pandas xgboost scikit-learn diff --git a/examples/advanced/sklearn-kmeans/requirements.txt b/examples/advanced/sklearn-kmeans/requirements.txt index aeafe651e1..22de3c503b 100644 --- a/examples/advanced/sklearn-kmeans/requirements.txt +++ b/examples/advanced/sklearn-kmeans/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc pandas scikit-learn joblib diff --git a/examples/advanced/sklearn-linear/requirements.txt b/examples/advanced/sklearn-linear/requirements.txt index aeafe651e1..22de3c503b 100644 --- a/examples/advanced/sklearn-linear/requirements.txt +++ b/examples/advanced/sklearn-linear/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc pandas scikit-learn joblib diff --git a/examples/advanced/sklearn-svm/requirements.txt b/examples/advanced/sklearn-svm/requirements.txt index aeafe651e1..22de3c503b 100644 --- a/examples/advanced/sklearn-svm/requirements.txt +++ b/examples/advanced/sklearn-svm/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc pandas scikit-learn joblib diff --git a/examples/advanced/swarm_learning/requirements.txt b/examples/advanced/swarm_learning/requirements.txt index 0804527963..3bbfea441b 100644 --- a/examples/advanced/swarm_learning/requirements.txt +++ b/examples/advanced/swarm_learning/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/examples/advanced/vertical_federated_learning/cifar10-splitnn/requirements.txt b/examples/advanced/vertical_federated_learning/cifar10-splitnn/requirements.txt index 58f9cf9de7..57c627000c 100644 --- a/examples/advanced/vertical_federated_learning/cifar10-splitnn/requirements.txt +++ b/examples/advanced/vertical_federated_learning/cifar10-splitnn/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.4.0 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/examples/advanced/vertical_xgboost/requirements.txt b/examples/advanced/vertical_xgboost/requirements.txt index 6bf2c8cbe1..a9a1d31eda 100644 --- a/examples/advanced/vertical_xgboost/requirements.txt +++ b/examples/advanced/vertical_xgboost/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc openmined.psi==1.1.1 pandas tensorboard diff --git a/examples/advanced/xgboost/histogram-based/requirements.txt b/examples/advanced/xgboost/histogram-based/requirements.txt index fcdcad4892..8311f62b9f 100644 --- a/examples/advanced/xgboost/histogram-based/requirements.txt +++ b/examples/advanced/xgboost/histogram-based/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc pandas xgboost>=2.0.0 scikit-learn diff --git a/examples/advanced/xgboost/tree-based/requirements.txt b/examples/advanced/xgboost/tree-based/requirements.txt index 02353a0589..96c88f1ec4 100644 --- a/examples/advanced/xgboost/tree-based/requirements.txt +++ b/examples/advanced/xgboost/tree-based/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc pandas xgboost scikit-learn diff --git a/examples/hello-world/hello-ccwf/requirements.txt b/examples/hello-world/hello-ccwf/requirements.txt new file mode 100644 index 0000000000..e4605852b5 --- /dev/null +++ b/examples/hello-world/hello-ccwf/requirements.txt @@ -0,0 +1 @@ +nvflare~=2.4.0rc diff --git a/examples/hello-world/hello-cyclic/requirements.txt b/examples/hello-world/hello-cyclic/requirements.txt index 0f8ce21d90..8f8b6bc27b 100644 --- a/examples/hello-world/hello-cyclic/requirements.txt +++ b/examples/hello-world/hello-cyclic/requirements.txt @@ -1,2 +1,2 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc tensorflow diff --git a/examples/hello-world/hello-numpy-cross-val/requirements.txt b/examples/hello-world/hello-numpy-cross-val/requirements.txt index 3fdbf10587..e4605852b5 100644 --- a/examples/hello-world/hello-numpy-cross-val/requirements.txt +++ b/examples/hello-world/hello-numpy-cross-val/requirements.txt @@ -1 +1 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc diff --git a/examples/hello-world/hello-numpy-sag/requirements.txt b/examples/hello-world/hello-numpy-sag/requirements.txt index 3fdbf10587..e4605852b5 100644 --- a/examples/hello-world/hello-numpy-sag/requirements.txt +++ b/examples/hello-world/hello-numpy-sag/requirements.txt @@ -1 +1 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc diff --git a/examples/hello-world/hello-pt/requirements.txt b/examples/hello-world/hello-pt/requirements.txt index 05e7ad671e..265102f82c 100644 --- a/examples/hello-world/hello-pt/requirements.txt +++ b/examples/hello-world/hello-pt/requirements.txt @@ -1 +1,3 @@ -nvflare[PT]>=2.3.0 +nvflare~=2.4.0rc +torch +torchvision diff --git a/examples/hello-world/hello-tf2/requirements.txt b/examples/hello-world/hello-tf2/requirements.txt index 0f8ce21d90..8f8b6bc27b 100644 --- a/examples/hello-world/hello-tf2/requirements.txt +++ b/examples/hello-world/hello-tf2/requirements.txt @@ -1,2 +1,2 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc tensorflow diff --git a/examples/hello-world/ml-to-fl/np/requirements.txt b/examples/hello-world/ml-to-fl/np/requirements.txt new file mode 100644 index 0000000000..e4605852b5 --- /dev/null +++ b/examples/hello-world/ml-to-fl/np/requirements.txt @@ -0,0 +1 @@ +nvflare~=2.4.0rc diff --git a/examples/hello-world/ml-to-fl/pt/requirements.txt b/examples/hello-world/ml-to-fl/pt/requirements.txt index 91244a6ec2..ea496a9976 100644 --- a/examples/hello-world/ml-to-fl/pt/requirements.txt +++ b/examples/hello-world/ml-to-fl/pt/requirements.txt @@ -1,3 +1,5 @@ -nvflare[PT]>=2.4.0 +nvflare~=2.4.0rc +torch +torchvision jsonargparse[signatures]>=4.17.0 pytorch_lightning diff --git a/examples/hello-world/ml-to-fl/tf/requirements.txt b/examples/hello-world/ml-to-fl/tf/requirements.txt index 573f902d09..8f8b6bc27b 100644 --- a/examples/hello-world/ml-to-fl/tf/requirements.txt +++ b/examples/hello-world/ml-to-fl/tf/requirements.txt @@ -1,2 +1,2 @@ -nvflare>=2.4.0 +nvflare~=2.4.0rc tensorflow diff --git a/examples/hello-world/step-by-step/cifar10/stats/requirements.txt b/examples/hello-world/step-by-step/cifar10/stats/requirements.txt index 45e20cc1ee..9e0a46f617 100644 --- a/examples/hello-world/step-by-step/cifar10/stats/requirements.txt +++ b/examples/hello-world/step-by-step/cifar10/stats/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc numpy monai[itk] pandas diff --git a/examples/hello-world/step-by-step/requirements.txt b/examples/hello-world/step-by-step/requirements.txt index e4bdfc07bf..3bbfea441b 100644 --- a/examples/hello-world/step-by-step/requirements.txt +++ b/examples/hello-world/step-by-step/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.2 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/integration/monai/examples/spleen_ct_segmentation_local/requirements.txt b/integration/monai/examples/spleen_ct_segmentation_local/requirements.txt index f1b948f162..716276ccd5 100644 --- a/integration/monai/examples/spleen_ct_segmentation_local/requirements.txt +++ b/integration/monai/examples/spleen_ct_segmentation_local/requirements.txt @@ -3,7 +3,7 @@ nibabel fire pytorch-ignite>=0.4.10 monai>=1.3.0 -nvflare>=2.3.0 +nvflare~=2.4.0rc monai_nvflare>=0.2.3 tensorboard mlflow diff --git a/integration/monai/examples/spleen_ct_segmentation_sim/requirements.txt b/integration/monai/examples/spleen_ct_segmentation_sim/requirements.txt index ff0407a24e..1b0abf8525 100644 --- a/integration/monai/examples/spleen_ct_segmentation_sim/requirements.txt +++ b/integration/monai/examples/spleen_ct_segmentation_sim/requirements.txt @@ -3,7 +3,7 @@ nibabel fire pytorch-ignite>=0.4.10 monai>=1.3.0 -nvflare>=2.3.0 +nvflare~=2.4.0rc monai_nvflare>=0.2.3 tensorboard scikit-image diff --git a/integration/nemo/examples/prompt_learning/requirements.txt b/integration/nemo/examples/prompt_learning/requirements.txt index 3fdbf10587..e4605852b5 100644 --- a/integration/nemo/examples/prompt_learning/requirements.txt +++ b/integration/nemo/examples/prompt_learning/requirements.txt @@ -1 +1 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc diff --git a/integration/sample/README.md b/integration/sample/README.md index cbe0a1917c..3a5322e40e 100644 --- a/integration/sample/README.md +++ b/integration/sample/README.md @@ -29,4 +29,4 @@ Every project and implementation has some assumptions behind it, either about th ## Required NVFLARE version -pip3 install nvflare>=2.3.0 +pip3 install nvflare>=2.4.0 diff --git a/research/auto-fed-rl/requirements.txt b/research/auto-fed-rl/requirements.txt index 0804527963..3bbfea441b 100644 --- a/research/auto-fed-rl/requirements.txt +++ b/research/auto-fed-rl/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/research/fed-ce/requirements.txt b/research/fed-ce/requirements.txt index b7dd1625cf..5757f5b0ea 100644 --- a/research/fed-ce/requirements.txt +++ b/research/fed-ce/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/research/fed-sm/requirements.txt b/research/fed-sm/requirements.txt index 56ed083280..71feadf756 100644 --- a/research/fed-sm/requirements.txt +++ b/research/fed-sm/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc websockets torch torchvision diff --git a/research/one-shot-vfl/requirements.txt b/research/one-shot-vfl/requirements.txt index 2f293a7a7c..0e0eac05ba 100644 --- a/research/one-shot-vfl/requirements.txt +++ b/research/one-shot-vfl/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc torch torchvision tensorboard diff --git a/research/quantifying-data-leakage/requirements.txt b/research/quantifying-data-leakage/requirements.txt index 060b0dd44c..b362ba016b 100644 --- a/research/quantifying-data-leakage/requirements.txt +++ b/research/quantifying-data-leakage/requirements.txt @@ -1,4 +1,4 @@ -nvflare>=2.3.0 +nvflare~=2.4.0rc pytorch-ignite>=0.4.10 torchvision monai>=1.0.1 From 186f186d6ee22815075caf610da758d7208eb1b8 Mon Sep 17 00:00:00 2001 From: Yuhong Wen Date: Tue, 30 Jan 2024 13:33:06 -0500 Subject: [PATCH 22/39] Removed the no need client custom path. (#2322) * Removed the no need client custom path. * Removed the commented out codes. --- nvflare/private/fed/app/simulator/simulator_runner.py | 3 --- 1 file changed, 3 deletions(-) diff --git a/nvflare/private/fed/app/simulator/simulator_runner.py b/nvflare/private/fed/app/simulator/simulator_runner.py index 78fdc93e59..b6a84c4516 100644 --- a/nvflare/private/fed/app/simulator/simulator_runner.py +++ b/nvflare/private/fed/app/simulator/simulator_runner.py @@ -331,9 +331,6 @@ def create_client(self, client_name): client_name, self.args ) self.federated_clients.append(client) - app_root = os.path.join(self.simulator_root, "app_" + client_name) - app_custom_folder = os.path.join(app_root, "custom") - sys.path.append(app_custom_folder) def _set_client_status(self): for client in self.federated_clients: From dd513f12686c07f1d99c81a00a0369244c779688 Mon Sep 17 00:00:00 2001 From: Isaac Yang Date: Wed, 31 Jan 2024 13:08:21 -0800 Subject: [PATCH 23/39] Enhance the handling of RC of task returned from clients --- nvflare/app_common/workflows/cyclic_ctl.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/nvflare/app_common/workflows/cyclic_ctl.py b/nvflare/app_common/workflows/cyclic_ctl.py index b274aa1f77..754e1b06b6 100644 --- a/nvflare/app_common/workflows/cyclic_ctl.py +++ b/nvflare/app_common/workflows/cyclic_ctl.py @@ -16,6 +16,7 @@ import random from nvflare.apis.client import Client +from nvflare.apis.fl_constant import ReturnCode from nvflare.apis.fl_context import FLContext from nvflare.apis.impl.controller import ClientTask, Controller, Task from nvflare.apis.shareable import Shareable @@ -145,6 +146,19 @@ def _get_relay_orders(self, fl_ctx: FLContext): return targets def _process_result(self, client_task: ClientTask, fl_ctx: FLContext): + result = client_task.result + rc = result.get_return_code() + client_name = client_task.client.name + + # Raise errors if ReturnCode is not OK. + if rc and rc != ReturnCode.OK: + self.system_panic( + f"Result from {client_name} is bad, error code: {rc}. " + f"{self.__class__.__name__} exiting at round {self._current_round}.", + fl_ctx=fl_ctx, + ) + return False + # submitted shareable is stored in client_task.result # we need to update task.data with that shareable so the next target # will get the updated shareable From 029815e558373cbfabbbcb3cf241cf9fecba00db Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Wed, 31 Jan 2024 16:55:40 -0800 Subject: [PATCH 24/39] Add xgboost to CI/CD (#2328) --- .../base/app/config/config_fed_client.json | 5 +- .../app/config/config_fed_server.json | 3 +- .../app/config/config_fed_client.json | 1 - .../app/config/config_fed_server.json | 3 +- .../xgboost/utils/prepare_job_config.py | 11 +- .../standalone_job/xgb_histogram_examples.yml | 44 +++++++ .../standalone_job/xgb_tree_examples.yml | 75 +++++++++++ tests/integration_test/src/example.py | 32 ++++- tests/integration_test/src/utils.py | 118 ++++++++++-------- tests/integration_test/test_configs.yml | 3 + 10 files changed, 234 insertions(+), 61 deletions(-) create mode 100644 tests/integration_test/data/test_configs/standalone_job/xgb_histogram_examples.yml create mode 100644 tests/integration_test/data/test_configs/standalone_job/xgb_tree_examples.yml diff --git a/examples/advanced/xgboost/histogram-based/jobs/base/app/config/config_fed_client.json b/examples/advanced/xgboost/histogram-based/jobs/base/app/config/config_fed_client.json index c456968c25..1d687b3a37 100755 --- a/examples/advanced/xgboost/histogram-based/jobs/base/app/config/config_fed_client.json +++ b/examples/advanced/xgboost/histogram-based/jobs/base/app/config/config_fed_client.json @@ -1,5 +1,6 @@ { "format_version": 2, + "num_rounds": 100, "executors": [ { "tasks": [ @@ -10,14 +11,14 @@ "name": "FedXGBHistogramExecutor", "args": { "data_loader_id": "dataloader", - "num_rounds": 100, + "num_rounds": "{num_rounds}", "early_stopping_rounds": 2, "xgb_params": { "max_depth": 8, "eta": 0.1, "objective": "binary:logistic", "eval_metric": "auc", - "tree_method": "gpu_hist", + "tree_method": "hist", "nthread": 16 } } diff --git a/examples/advanced/xgboost/tree-based/jobs/bagging_base/app/config/config_fed_server.json b/examples/advanced/xgboost/tree-based/jobs/bagging_base/app/config/config_fed_server.json index 124a2296f9..f35526c721 100755 --- a/examples/advanced/xgboost/tree-based/jobs/bagging_base/app/config/config_fed_server.json +++ b/examples/advanced/xgboost/tree-based/jobs/bagging_base/app/config/config_fed_server.json @@ -1,5 +1,6 @@ { "format_version": 2, + "num_rounds": 101, "server": { "heart_beat_timeout": 600, @@ -34,7 +35,7 @@ "name": "ScatterAndGather", "args": { "min_clients": 5, - "num_rounds": 101, + "num_rounds": "{num_rounds}", "start_round": 0, "wait_time_after_min_received": 0, "aggregator_id": "aggregator", diff --git a/examples/advanced/xgboost/tree-based/jobs/cyclic_base/app/config/config_fed_client.json b/examples/advanced/xgboost/tree-based/jobs/cyclic_base/app/config/config_fed_client.json index 319c26de1a..6b25f996bb 100755 --- a/examples/advanced/xgboost/tree-based/jobs/cyclic_base/app/config/config_fed_client.json +++ b/examples/advanced/xgboost/tree-based/jobs/cyclic_base/app/config/config_fed_client.json @@ -13,7 +13,6 @@ "data_loader_id": "dataloader", "training_mode": "cyclic", "num_client_bagging": 1, - "lr_mode": "scaled", "local_model_path": "model.json", "global_model_path": "model_global.json", "learning_rate": 0.1, diff --git a/examples/advanced/xgboost/tree-based/jobs/cyclic_base/app/config/config_fed_server.json b/examples/advanced/xgboost/tree-based/jobs/cyclic_base/app/config/config_fed_server.json index 686042e1b5..3f331b862c 100755 --- a/examples/advanced/xgboost/tree-based/jobs/cyclic_base/app/config/config_fed_server.json +++ b/examples/advanced/xgboost/tree-based/jobs/cyclic_base/app/config/config_fed_server.json @@ -1,5 +1,6 @@ { "format_version": 2, + "num_rounds": 20, "server": { "heart_beat_timeout": 600, @@ -29,7 +30,7 @@ "id": "cyclic_ctl", "name": "CyclicController", "args": { - "num_rounds": 20, + "num_rounds": "{num_rounds}", "task_assignment_timeout": 60, "persistor_id": "persistor", "shareable_generator_id": "shareable_generator", diff --git a/examples/advanced/xgboost/utils/prepare_job_config.py b/examples/advanced/xgboost/utils/prepare_job_config.py index f970faa016..28e5652a63 100644 --- a/examples/advanced/xgboost/utils/prepare_job_config.py +++ b/examples/advanced/xgboost/utils/prepare_job_config.py @@ -20,6 +20,8 @@ from nvflare.apis.fl_constant import JobConstants +SCRIPT_PATH = pathlib.Path(os.path.realpath(__file__)) +XGB_EXAMPLE_ROOT = SCRIPT_PATH.parent.parent.absolute() JOB_CONFIGS_ROOT = "jobs" MODE_ALGO_MAP = {"bagging": "tree-based", "cyclic": "tree-based", "histogram": "histogram-based"} @@ -84,7 +86,7 @@ def _get_src_job_dir(training_mode): "cyclic": "cyclic_base", "histogram": "base", } - return pathlib.Path(MODE_ALGO_MAP[training_mode]) / JOB_CONFIGS_ROOT / base_job_map[training_mode] + return XGB_EXAMPLE_ROOT / MODE_ALGO_MAP[training_mode] / JOB_CONFIGS_ROOT / base_job_map[training_mode] def _gen_deploy_map(num_sites: int, site_name_prefix: str) -> dict: @@ -133,6 +135,7 @@ def _update_client_config(config: dict, args, lr_scale, site_name: str): num_client_bagging = args.site_num config["executors"][0]["executor"]["args"]["num_client_bagging"] = num_client_bagging else: + config["num_rounds"] = args.round_num config["components"][0]["args"]["data_split_filename"] = data_split_name config["executors"][0]["executor"]["args"]["xgb_params"]["nthread"] = args.nthread config["executors"][0]["executor"]["args"]["xgb_params"]["tree_method"] = args.tree_method @@ -140,10 +143,10 @@ def _update_client_config(config: dict, args, lr_scale, site_name: str): def _update_server_config(config: dict, args): if args.training_mode == "bagging": - config["workflows"][0]["args"]["num_rounds"] = args.round_num + 1 + config["num_rounds"] = args.round_num + 1 config["workflows"][0]["args"]["min_clients"] = args.site_num elif args.training_mode == "cyclic": - config["workflows"][0]["args"]["num_rounds"] = int(args.round_num / args.site_num) + config["num_rounds"] = int(args.round_num / args.site_num) def _copy_custom_files(src_job_path, src_app_name, dst_job_path, dst_app_name): @@ -198,7 +201,7 @@ def main(): src_job_path = _get_src_job_dir(args.training_mode) # create a new job - dst_job_path = pathlib.Path(MODE_ALGO_MAP[args.training_mode]) / JOB_CONFIGS_ROOT / job_name + dst_job_path = XGB_EXAMPLE_ROOT / MODE_ALGO_MAP[args.training_mode] / JOB_CONFIGS_ROOT / job_name if not os.path.exists(dst_job_path): os.makedirs(dst_job_path) diff --git a/tests/integration_test/data/test_configs/standalone_job/xgb_histogram_examples.yml b/tests/integration_test/data/test_configs/standalone_job/xgb_histogram_examples.yml new file mode 100644 index 0000000000..38f3cd71ec --- /dev/null +++ b/tests/integration_test/data/test_configs/standalone_job/xgb_histogram_examples.yml @@ -0,0 +1,44 @@ +n_servers: 1 +n_clients: 2 +additional_python_paths: +- ../../examples/advanced/xgboost +cleanup: true +jobs_root_dir: ../../examples/advanced/xgboost/histogram-based/jobs + + +tests: +- test_name: Test a simplified copy of job higgs_2_histogram_uniform_split_uniform_lr + for xgboost histogram-based example. + event_sequence: + - actions: + - submit_job higgs_2_histogram_uniform_split_uniform_lr_copy + result: + type: job_submit_success + trigger: + data: Server started + type: server_log + - actions: + - ensure_current_job_done + result: + data: + run_finished: true + type: run_state + trigger: + data: + run_finished: true + type: run_state + setup: + - cp ../../examples/advanced/xgboost/histogram-based/requirements.txt + ../../examples/advanced/xgboost/histogram-based/temp_requirements.txt + - sed -i '/nvflare\|jupyter\|notebook/d' ../../examples/advanced/xgboost/histogram-based/temp_requirements.txt + - pip install -r ../../examples/advanced/xgboost/histogram-based/temp_requirements.txt + - bash ../../examples/advanced/xgboost/histogram-based/prepare_data.sh + - python3 ../../examples/advanced/xgboost/utils/prepare_job_config.py --site_num 2 --training_mode histogram + --split_method uniform --lr_mode uniform --nthread 16 --tree_method hist + - python3 convert_to_test_job.py + --job ../../examples/advanced/xgboost/histogram-based/jobs/higgs_2_histogram_uniform_split_uniform_lr + --post _copy + - rm -f ../../examples/advanced/xgboost/histogram-based/temp_requirements.txt + teardown: + - rm -rf ../../examples/advanced/xgboost/histogram-based/jobs/higgs_2_histogram_uniform_split_uniform_lr + - rm -rf ../../examples/advanced/xgboost/histogram-based/jobs/higgs_2_histogram_uniform_split_uniform_lr_copy diff --git a/tests/integration_test/data/test_configs/standalone_job/xgb_tree_examples.yml b/tests/integration_test/data/test_configs/standalone_job/xgb_tree_examples.yml new file mode 100644 index 0000000000..145b6aecf0 --- /dev/null +++ b/tests/integration_test/data/test_configs/standalone_job/xgb_tree_examples.yml @@ -0,0 +1,75 @@ +n_servers: 1 +n_clients: 5 +additional_python_paths: +- ../../examples/advanced/xgboost +cleanup: true +jobs_root_dir: ../../examples/advanced/xgboost/tree-based/jobs + + +tests: +- test_name: Test a simplified copy of job higgs_5_cyclic_uniform_split_uniform_lr + for xgboost tree-based example. + event_sequence: + - actions: + - submit_job higgs_5_cyclic_uniform_split_uniform_lr_copy + result: + type: job_submit_success + trigger: + data: Server started + type: server_log + - actions: + - ensure_current_job_done + result: + data: + run_finished: true + type: run_state + trigger: + data: + run_finished: true + type: run_state + setup: + - cp ../../examples/advanced/xgboost/tree-based/requirements.txt + ../../examples/advanced/xgboost/tree-based/temp_requirements.txt + - sed -i '/nvflare\|jupyter\|notebook/d' ../../examples/advanced/xgboost/tree-based/temp_requirements.txt + - pip install -r ../../examples/advanced/xgboost/tree-based/temp_requirements.txt + - bash ../../examples/advanced/xgboost/tree-based/prepare_data.sh + - python3 ../../examples/advanced/xgboost/utils/prepare_job_config.py --site_num 5 --training_mode cyclic + --split_method uniform --lr_mode uniform --nthread 16 --tree_method hist + - python3 convert_to_test_job.py + --job ../../examples/advanced/xgboost/tree-based/jobs/higgs_5_cyclic_uniform_split_uniform_lr + --post _copy + - rm -f ../../examples/advanced/xgboost/tree-based/temp_requirements.txt + teardown: + - rm -rf ../../examples/advanced/xgboost/tree-based/jobs/higgs_5_cyclic_uniform_split_uniform_lr + - rm -rf ../../examples/advanced/xgboost/tree-based/jobs/higgs_5_cyclic_uniform_split_uniform_lr_copy + +- test_name: Test a simplified copy of job higgs_5_bagging_uniform_split_uniform_lr + for xgboost tree-based example. + event_sequence: + - actions: + - submit_job higgs_5_bagging_uniform_split_uniform_lr_copy + result: + type: job_submit_success + trigger: + data: Server started + type: server_log + - actions: + - ensure_current_job_done + result: + data: + run_finished: true + type: run_state + trigger: + data: + run_finished: true + type: run_state + setup: + - python3 ../../examples/advanced/xgboost/utils/prepare_job_config.py --site_num 5 --training_mode bagging + --split_method uniform --lr_mode uniform --nthread 16 --tree_method hist + - python3 convert_to_test_job.py + --job ../../examples/advanced/xgboost/tree-based/jobs/higgs_5_bagging_uniform_split_uniform_lr + --post _copy + - rm -f ../../examples/advanced/xgboost/tree-based/temp_requirements.txt + teardown: + - rm -rf ../../examples/advanced/xgboost/tree-based/jobs/higgs_5_bagging_uniform_split_uniform_lr + - rm -rf ../../examples/advanced/xgboost/tree-based/jobs/higgs_5_bagging_uniform_split_uniform_lr_copy diff --git a/tests/integration_test/src/example.py b/tests/integration_test/src/example.py index f46d01b53c..58dbeeea35 100644 --- a/tests/integration_test/src/example.py +++ b/tests/integration_test/src/example.py @@ -17,7 +17,7 @@ class Example: - """This class represents a standardized example structure in NVFlare.""" + """This class represents a standardized example folder structure in NVFlare.""" def __init__( self, @@ -27,9 +27,37 @@ def __init__( additional_python_path: Optional[str] = None, prepare_data_script: Optional[str] = None, ): + """Constructor of Example. + + A standardized example folder looks like the following: + + .. code-block + + ./[example_root] + ./[jobs_folder_in_example] + ./job_name1 + ./job_name2 + ./job_name3 + ./[requirements] + ./[prepare_data_script] + + For example: + + .. code-block + + ./cifar10-sim + ./jobs + ./cifar10_central + ./cifar10_fedavg + ./cifar10_fedopt + ... + ./requirements.txt + ./prepare_data.sh + + """ self.root = os.path.abspath(root) if not os.path.exists(self.root): - raise FileNotFoundError("Example root directory does not exist.") + raise FileNotFoundError("Example's root directory does not exist.") self.name = os.path.basename(self.root) diff --git a/tests/integration_test/src/utils.py b/tests/integration_test/src/utils.py index ac0d09291e..0a945607c9 100644 --- a/tests/integration_test/src/utils.py +++ b/tests/integration_test/src/utils.py @@ -292,6 +292,7 @@ def _replace_config_fed_client(client_json_path: str): with open(client_json_path, "r+") as f: config_fed_client = json.load(f) config_fed_client["TRAIN_SPLIT_ROOT"] = "/tmp/nvflare/test_data" + config_fed_client["num_rounds"] = 2 config_fed_client["AGGREGATION_EPOCHS"] = 1 f.seek(0) json.dump(config_fed_client, f, indent=4) @@ -318,67 +319,84 @@ def simplify_job(job_folder_path: str, postfix: str = POSTFIX): def generate_test_config_yaml_for_example( example: Example, project_yaml: str = PROJECT_YAML, - postfix: str = POSTFIX, + job_postfix: str = POSTFIX, ) -> List[str]: - """Generates test configuration yaml for NVFlare example. + """Generates test configurations for an NVFlare example folder. Args: - example: A well-formatted NVFlare example. - project_yaml: Project yaml file for the testing of this example. - postfix: Postfix for the newly generated job. + example (Example): A well-formatted NVFlare example folder. + project_yaml (str): Project yaml file for the testing of this example. + job_postfix (str): Postfix for the newly generated job. """ - output_yamls = [] os.makedirs(OUTPUT_YAML_DIR, exist_ok=True) for job in os.listdir(example.jobs_root_dir): - output_yaml = os.path.join(OUTPUT_YAML_DIR, f"{example.name}_{job}.yml") - job_dir = os.path.join(example.jobs_root_dir, job) - requirements_file = os.path.join(example.root, example.requirements_file) - new_requirements_file = os.path.join(example.root, "temp_requirements.txt") - exclude_requirements = "\\|".join(REQUIREMENTS_TO_EXCLUDE) - - setup = [ - f"cp {requirements_file} {new_requirements_file}", - f"sed -i '/{exclude_requirements}/d' {new_requirements_file}", - f"pip install -r {new_requirements_file}", - ] - if example.prepare_data_script is not None: - setup.append(f"bash {example.prepare_data_script}") - setup.append(f"python convert_to_test_job.py --job {job_dir} --post {postfix}") - setup.append(f"rm -f {new_requirements_file}") - - config = { - "ha": True, - "jobs_root_dir": example.jobs_root_dir, - "cleanup": True, - "project_yaml": project_yaml, - "additional_python_paths": example.additional_python_paths, - "tests": [ - { - "test_name": f"Test a simplified copy of job {job} for example {example.name}.", - "event_sequence": [ - { - "trigger": {"type": "server_log", "data": "Server started"}, - "actions": [f"submit_job {job}{postfix}"], - "result": {"type": "job_submit_success"}, - }, - { - "trigger": {"type": "run_state", "data": {"run_finished": True}}, - "actions": ["ensure_current_job_done"], - "result": {"type": "run_state", "data": {"run_finished": True}}, - }, - ], - "setup": setup, - "teardown": [f"rm -rf {job_dir}{postfix}"], - } - ], - } - with open(output_yaml, "w") as yaml_file: - yaml.dump(config, yaml_file, default_flow_style=False) + output_yaml = _generate_test_config_for_one_job(example, job, project_yaml, job_postfix) output_yamls.append(output_yaml) return output_yamls +def _generate_test_config_for_one_job( + example: Example, + job: str, + project_yaml: str = PROJECT_YAML, + postfix: str = POSTFIX, +) -> str: + """Generates test configuration yaml for an NVFlare example. + + Args: + example (Example): A well-formatted NVFlare example. + job (str): name of the job. + project_yaml (str): Project yaml file for the testing of this example. + postfix (str): Postfix for the newly generated job. + """ + output_yaml = os.path.join(OUTPUT_YAML_DIR, f"{example.name}_{job}.yml") + job_dir = os.path.join(example.jobs_root_dir, job) + requirements_file = os.path.join(example.root, example.requirements_file) + new_requirements_file = os.path.join(example.root, "temp_requirements.txt") + exclude_requirements = "\\|".join(REQUIREMENTS_TO_EXCLUDE) + + setup = [ + f"cp {requirements_file} {new_requirements_file}", + f"sed -i '/{exclude_requirements}/d' {new_requirements_file}", + f"pip install -r {new_requirements_file}", + ] + if example.prepare_data_script is not None: + setup.append(f"bash {example.prepare_data_script}") + setup.append(f"python convert_to_test_job.py --job {job_dir} --post {postfix}") + setup.append(f"rm -f {new_requirements_file}") + + config = { + "ha": True, + "jobs_root_dir": example.jobs_root_dir, + "cleanup": True, + "project_yaml": project_yaml, + "additional_python_paths": example.additional_python_paths, + "tests": [ + { + "test_name": f"Test a simplified copy of job {job} for example {example.name}.", + "event_sequence": [ + { + "trigger": {"type": "server_log", "data": "Server started"}, + "actions": [f"submit_job {job}{postfix}"], + "result": {"type": "job_submit_success"}, + }, + { + "trigger": {"type": "run_state", "data": {"run_finished": True}}, + "actions": ["ensure_current_job_done"], + "result": {"type": "run_state", "data": {"run_finished": True}}, + }, + ], + "setup": setup, + "teardown": [f"rm -rf {job_dir}{postfix}"], + } + ], + } + with open(output_yaml, "w") as yaml_file: + yaml.dump(config, yaml_file, default_flow_style=False) + return output_yaml + + def _read_admin_json_file(admin_json_file) -> dict: if not os.path.exists(admin_json_file): raise RuntimeError("Missing admin json file.") diff --git a/tests/integration_test/test_configs.yml b/tests/integration_test/test_configs.yml index a179a6ce79..d75f59a784 100644 --- a/tests/integration_test/test_configs.yml +++ b/tests/integration_test/test_configs.yml @@ -29,3 +29,6 @@ test_configs: - ./data/test_configs/standalone_job/cifar_examples.yml stats: - ./data/test_configs/standalone_job/image_stats.yml + xgboost: + - ./data/test_configs/standalone_job/xgb_histogram_examples.yml + - ./data/test_configs/standalone_job/xgb_tree_examples.yml From edaf8d200a5eab10465d85a9ad0d0df5bb070f82 Mon Sep 17 00:00:00 2001 From: Holger Roth <6304754+holgerroth@users.noreply.github.com> Date: Thu, 1 Feb 2024 11:10:49 -0500 Subject: [PATCH 25/39] update cifar10 and gnn examples (#2340) --- .../cifar10-sim/figs/plot_tensorboard_events.py | 7 +++++-- .../advanced/cifar10/cifar10-sim/run_simulator.sh | 7 +------ examples/advanced/gnn/README.md | 12 ++++++------ nvflare/app_common/workflows/base_fedavg.py | 6 +++++- nvflare/app_common/workflows/model_controller.py | 10 ++++++++-- 5 files changed, 25 insertions(+), 17 deletions(-) diff --git a/examples/advanced/cifar10/cifar10-sim/figs/plot_tensorboard_events.py b/examples/advanced/cifar10/cifar10-sim/figs/plot_tensorboard_events.py index 495dadf4c2..0e3c06a468 100644 --- a/examples/advanced/cifar10/cifar10-sim/figs/plot_tensorboard_events.py +++ b/examples/advanced/cifar10/cifar10-sim/figs/plot_tensorboard_events.py @@ -30,7 +30,7 @@ # 4.1 Central vs. FedAvg experiments = { - "cifar10_central": {"tag": "val_acc_local_model"}, + "cifar10_central": {"tag": "val_acc_local_model", "alpha": 0.0}, "cifar10_fedavg": {"tag": "val_acc_global_model", "alpha": 1.0}, } @@ -95,6 +95,8 @@ def main(): alpha = exp.get("alpha", None) if alpha: config_name = config_name + f"*alpha{alpha}" + else: + raise ValueError(f"Expected an alpha value to be provided but got alpha={alpha}") eventfile = glob.glob( os.path.join(client_results_root, config_name, "**", "app_site-1", "events.*"), recursive=True ) @@ -116,7 +118,8 @@ def main(): try: xsite_data[k].append(xsite_results["site-1"][k]["val_accuracy"]) except Exception as e: - raise ValueError(f"No val_accuracy for {k} in {xsite_file}!") + xsite_data[k].append(None) + print(f"Warning: No val_accuracy for {k} in {xsite_file}!") print("Training TB data:") print(pd.DataFrame(data)) diff --git a/examples/advanced/cifar10/cifar10-sim/run_simulator.sh b/examples/advanced/cifar10/cifar10-sim/run_simulator.sh index 24a901a4ae..2fdd6eca45 100755 --- a/examples/advanced/cifar10/cifar10-sim/run_simulator.sh +++ b/examples/advanced/cifar10/cifar10-sim/run_simulator.sh @@ -8,12 +8,7 @@ n_clients=$4 # specify output workdir RESULT_ROOT=/tmp/nvflare/sim_cifar10 -if [ 1 -eq "$(echo "${alpha} > 0" | bc)" ] -then - out_workspace=${RESULT_ROOT}/${job}_alpha${alpha} -else - out_workspace=${RESULT_ROOT}/${job} -fi +out_workspace=${RESULT_ROOT}/${job}_alpha${alpha} # run FL simulator ./set_alpha.sh "${job}" "${alpha}" diff --git a/examples/advanced/gnn/README.md b/examples/advanced/gnn/README.md index 3a743431ce..f8c0c415ea 100644 --- a/examples/advanced/gnn/README.md +++ b/examples/advanced/gnn/README.md @@ -31,7 +31,7 @@ python3 -m pip install -r requirements.txt ``` To support functions of PyTorch Geometric necessary for this example, we need extra dependencies. Please refer to [installation guide](https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html) and install accordingly: ``` -pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.1.0+cpu.html +python3 -m pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.1.0+cpu.html ``` #### Job Template @@ -46,8 +46,8 @@ nvflare job list_templates We can see the "sag_gnn" template is available #### Protein Classification -The PPI dataset is directly available via torch_geometric library, we randomly split the dataset to 2 subsets, one for each client. -First, we run the local training on each client, as well as the whole dataset. +The PPI dataset is directly available via torch_geometric library, we randomly split the dataset to 2 subsets, one for each client (`--client_id 1` and `--client_id 2`). +First, we run the local training on each client, as well as the whole dataset with `--client_id 0`. ``` python3 code/graphsage_protein_local.py --client_id 0 python3 code/graphsage_protein_local.py --client_id 1 @@ -64,7 +64,7 @@ For client configs, we set client_ids for each client, and the number of local e For server configs, we set the number of rounds for federated training, the key metric for model selection, and the model class path with model hyperparameters. -With the produced job, we run the federated training on both clients via FedAvg using NVFlare Simulator. +With the produced job, we run the federated training on both clients via FedAvg using the NVFlare Simulator. ``` nvflare simulator -w /tmp/nvflare/gnn/protein_fl_workspace -n 2 -t 2 /tmp/nvflare/jobs/gnn_protein ``` @@ -74,7 +74,7 @@ We first download the Elliptic++ dataset to `/tmp/nvflare/datasets/elliptic_pp` - `txs_classes.csv`: transaction id and its class (licit or illicit) - `txs_edgelist.csv`: connections for transaction ids - `txs_features.csv`: transaction id and its features -Then, we run the local training on each client, as well as the whole dataset. +Then, we run the local training on each client, as well as the whole dataset. Again, `--client_id 0` uses all data. ``` python3 code/graphsage_finance_local.py --client_id 0 python3 code/graphsage_finance_local.py --client_id 1 @@ -87,7 +87,7 @@ nvflare job create -force -j "/tmp/nvflare/jobs/gnn_finance" -w "sag_gnn" -sd "c -f app_2/config_fed_client.conf app_script="graphsage_finance_fl.py" app_config="--client_id 2 --epochs 10" \ -f app_server/config_fed_server.conf num_rounds=7 key_metric="validation_auc" model_class_path="pyg_sage.SAGE" components[0].args.model.args.in_channels=165 components[0].args.model.args.hidden_channels=256 components[0].args.model.args.num_layers=3 components[0].args.model.args.num_classes=2 ``` -And with the produced job, we run the federated training on both clients via FedAvg using NVFlare Simulator. +And with the produced job, we run the federated training on both clients via FedAvg using the NVFlare Simulator. ``` nvflare simulator -w /tmp/nvflare/gnn/finance_fl_workspace -n 2 -t 2 /tmp/nvflare/jobs/gnn_finance ``` diff --git a/nvflare/app_common/workflows/base_fedavg.py b/nvflare/app_common/workflows/base_fedavg.py index d031998e35..cd196d86b3 100644 --- a/nvflare/app_common/workflows/base_fedavg.py +++ b/nvflare/app_common/workflows/base_fedavg.py @@ -17,6 +17,7 @@ from nvflare.apis.fl_constant import FLMetaKey from nvflare.app_common.abstract.fl_model import FLModel +from nvflare.app_common.abstract.model import make_model_learnable from nvflare.app_common.aggregators.weighted_aggregation_helper import WeightedAggregationHelper from nvflare.app_common.app_constant import AppConstants from nvflare.app_common.app_event_type import AppEventType @@ -142,5 +143,8 @@ def update_model(self, aggr_result): self.model = FLModelUtils.update_model(self.model, aggr_result) - self.fl_ctx.set_prop(AppConstants.GLOBAL_MODEL, self.model, private=True, sticky=True) + # persistor uses Learnable format to save model + ml = make_model_learnable(weights=self.model.params, meta_props=self.model.meta) + self.fl_ctx.set_prop(AppConstants.GLOBAL_MODEL, ml, private=True, sticky=True) + self.event(AppEventType.AFTER_SHAREABLE_TO_LEARNABLE) diff --git a/nvflare/app_common/workflows/model_controller.py b/nvflare/app_common/workflows/model_controller.py index 5320a432be..0b65f07539 100644 --- a/nvflare/app_common/workflows/model_controller.py +++ b/nvflare/app_common/workflows/model_controller.py @@ -138,7 +138,9 @@ def start_controller(self, fl_ctx: FLContext) -> None: else: self.model = FLModel(params_type=ParamsType.FULL, params={}) - self.fl_ctx.set_prop(AppConstants.GLOBAL_MODEL, self.model, private=True, sticky=True) + # persistor uses Learnable format to save model + ml = make_model_learnable(weights=self.model.params, meta_props=self.model.meta) + self.fl_ctx.set_prop(AppConstants.GLOBAL_MODEL, ml, private=True, sticky=True) self.event(AppEventType.INITIAL_MODEL_LOADED) self.engine = self.fl_ctx.get_engine() @@ -231,7 +233,11 @@ def _process_result(self, client_task: ClientTask, fl_ctx: FLContext) -> None: result = client_task.result client_name = client_task.client.name + self.fl_ctx.set_prop(AppConstants.CURRENT_ROUND, self._current_round, private=True, sticky=True) + + self.event(AppEventType.BEFORE_CONTRIBUTION_ACCEPT) self._accept_train_result(client_name=client_name, result=result, fl_ctx=fl_ctx) + self.event(AppEventType.AFTER_CONTRIBUTION_ACCEPT) # Turn result into FLModel result_model = FLModelUtils.from_shareable(result) @@ -270,7 +276,6 @@ def _accept_train_result(self, client_name: str, result: Shareable, fl_ctx: FLCo ) return - self.fl_ctx.set_prop(AppConstants.CURRENT_ROUND, self._current_round, private=True, sticky=True) self.fl_ctx.set_prop(AppConstants.TRAINING_RESULT, result, private=True, sticky=False) @abstractmethod @@ -307,6 +312,7 @@ def save_model(self): ) or self._current_round == self._num_rounds - 1: self.info("Start persist model on server.") self.event(AppEventType.BEFORE_LEARNABLE_PERSIST) + # persistor uses Learnable format to save model ml = make_model_learnable(weights=self.model.params, meta_props=self.model.meta) self.persistor.save(ml, self.fl_ctx) self.event(AppEventType.AFTER_LEARNABLE_PERSIST) From da9562a69422feddd7aeebbfac0daba7af76a53c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Thu, 1 Feb 2024 20:13:51 -0800 Subject: [PATCH 26/39] Address final VDR feedbacks (#2332) * Address final VDR feedbacks * Fix typo * Address feedback --- .../fl_experiment_tracking_mlflow.rst | 12 +- docs/programming_guide.rst | 1 + docs/programming_guide/execution_api_type.rst | 29 +- .../3rd_party_integration.rst | 268 +++++++----------- .../execution_api_type/client_api.rst | 2 +- docs/release_notes/flare_240.rst | 2 +- docs/resources/3rd_party_trainer.py | 59 ++++ ...fed_client.json => config_fed_client.conf} | 0 ...fed_server.json => config_fed_server.conf} | 0 ...fed_client.json => config_fed_client.conf} | 0 ...fed_server.json => config_fed_server.conf} | 0 examples/hello-world/ml-to-fl/np/README.md | 23 +- examples/hello-world/ml-to-fl/pt/README.md | 37 ++- .../pt/code/cifar10_lightning_ddp_fl.py | 4 +- examples/hello-world/ml-to-fl/tf/README.md | 30 +- ...fed_client.json => config_fed_client.conf} | 0 ...fed_server.json => config_fed_server.conf} | 4 +- nvflare/client/lightning/__init__.py | 10 + nvflare/lighter/dummy_project.yml | 7 +- setup.cfg | 4 + 20 files changed, 284 insertions(+), 208 deletions(-) create mode 100644 docs/resources/3rd_party_trainer.py rename examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/{config_fed_client.json => config_fed_client.conf} (100%) rename examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/{config_fed_server.json => config_fed_server.conf} (100%) rename examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/{config_fed_client.json => config_fed_client.conf} (100%) rename examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/{config_fed_server.json => config_fed_server.conf} (100%) rename integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/{config_fed_client.json => config_fed_client.conf} (100%) rename integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/{config_fed_server.json => config_fed_server.conf} (95%) diff --git a/docs/examples/fl_experiment_tracking_mlflow.rst b/docs/examples/fl_experiment_tracking_mlflow.rst index 14b6e36860..d9a9b63891 100644 --- a/docs/examples/fl_experiment_tracking_mlflow.rst +++ b/docs/examples/fl_experiment_tracking_mlflow.rst @@ -53,10 +53,10 @@ Adding MLflow Logging to Configurations Inside the config folder there are two files, ``config_fed_client.json`` and ``config_fed_server.json``. -.. literalinclude:: ../../examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_client.json - :language: json +.. literalinclude:: ../../examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_client.conf + :language: :linenos: - :caption: config_fed_client.json + :caption: config_fed_client.conf Take a look at the components section of the client config at line 24. The first component is the ``pt_learner`` which contains the initialization, training, and validation logic. @@ -69,10 +69,10 @@ within NVFlare with the information to track. Finally, :class:`ConvertToFedEvent` converts local events to federated events. This changes the event ``analytix_log_stats`` into a fed event ``fed.analytix_log_stats``, which will then be streamed from the clients to the server. -.. literalinclude:: ../../examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_server.json - :language: json +.. literalinclude:: ../../examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_server.conf + :language: :linenos: - :caption: config_fed_server.json + :caption: config_fed_server.conf Under the component section in the server config, we have the :class:`MLflowReceiver`. This component receives diff --git a/docs/programming_guide.rst b/docs/programming_guide.rst index 835839245f..28e8b7992b 100644 --- a/docs/programming_guide.rst +++ b/docs/programming_guide.rst @@ -37,6 +37,7 @@ Please refer to :ref:`application` for more details. programming_guide/workflows_and_controllers programming_guide/execution_api_type + programming_guide/fl_model programming_guide/shareable programming_guide/data_exchange_object programming_guide/fl_context diff --git a/docs/programming_guide/execution_api_type.rst b/docs/programming_guide/execution_api_type.rst index 1095368afc..77baf7806a 100644 --- a/docs/programming_guide/execution_api_type.rst +++ b/docs/programming_guide/execution_api_type.rst @@ -1,11 +1,12 @@ .. _execution_api_type: -################## -Execution API Type -################## +####################### +From Local to Federated +####################### In the FLARE system, a federated learning algorithm is defined in a Job format (for details, please refer to :ref:`job`). + A Job consists of multiple "workflows" and "executors." The simplified job execution flow is as follows: @@ -16,15 +17,19 @@ The simplified job execution flow is as follows: - If it is not done, it schedules a new task - If it is done, it proceeds to the next workflow in the Job. -Users need to adapt their local training logic into FLARE's task execution -abstractions to make their training federated. +Users need to adapt their local training or computing logic into FLARE's task +execution abstractions to make their training or computing federated. We offer various levels of abstraction for writing task execution code, catering to use cases that span from complete customizability to easy user adaptation. +Execution API Type +================== + Below is a general overview of the key ideas and use cases for each type: -**Client API** +Client API +---------- The :ref:`client_api` provides the most straightforward way to write FL code, and can easily be used to convert centralized code with minimal code changes. @@ -32,9 +37,11 @@ The Client API uses the :class:`FLModel` - :class:`FlareAgentWithCellPipe` -You can create the FlareAgent as the following code: +You can create the FlareAgentWithCellPipe as the following code: .. code-block:: python @@ -72,7 +72,7 @@ You can create the FlareAgent as the following code: agent = FlareAgentWithCellPipe( root_url="grpc://server:8002", - flare_site_name=args.site_name, + site_name=args.site_name, agent_id=args.agent_id, workspace_dir=args.workspace, secure_mode=True, @@ -142,72 +142,15 @@ If this call is missed, the program may not exit properly. agent.stop() -Putting Together ----------------- +5. Putting Together +------------------- Now we learn all the necessary steps, we can put together into the following example code of this usage pattern: -.. code-block:: python - - import argparse - import logging - - from nvflare.client.defs import RC, AgentClosed, MetaKey - from nvflare.client.flare_agent import FlareAgentWithCellPipe - - NUMPY_KEY = "numpy_key" - - - def main(): - - logging.basicConfig() - logging.getLogger().setLevel(logging.INFO) - - parser = argparse.ArgumentParser() - parser.add_argument("--workspace", "-w", type=str, help="workspace folder", required=False, default=".") - parser.add_argument("--site_name", "-s", type=str, help="flare site name", required=True) - parser.add_argument("--agent_id", "-a", type=str, help="agent id", required=True) - - args = parser.parse_args() +.. literalinclude:: ../../resources/3rd_party_trainer.py + :language: python - # 1. create the agent - agent = FlareAgentWithCellPipe( - root_url="grpc://server:8002", - flare_site_name=args.site_name, - agent_id=args.agent_id, - workspace_dir=args.workspace, - secure_mode=True, - submit_result_timeout=2.0, - heartbeat_timeout=120.0, - ) - - # 2. start the agent - agent.start() - - # 3. processing tasks - while True: - print("getting task ...") - try: - task = agent.get_task() - except AgentClosed: - print("agent closed - exit") - break - - print(f"got task: {task}") - rc, meta, result = train(task.data) # perform train task - submitted = agent.submit_result(TaskResult(data=result, meta=meta, return_code=rc)) - print(f"result submitted: {submitted}") - - # 4. stop the agent - agent.stop() - - - def train(model): - ... - - if __name__ == "__main__": - main() Notes: @@ -250,81 +193,61 @@ An example looks like: - name: site_1 type: client org: nvidia - listening_host: site_1.maglev.nvidia.com + listening_host: localhost - name: site_2 type: client org: nvidia - listening_host: site_2.maglev.nvidia.com + listening_host: localhost Once the project is provisioned, check the "startup" kit generated for the clients. You should see the following files, among others: client.crt, client.key, server.crt, server.key, rootCA.pem -Note that the specified listening_port of a site must be accessible to the trainer of the site. - -Step Two - Setup for Adhoc Direct Connection between FL Client and Trainer --------------------------------------------------------------------------- - -FL client and the trainer can always talk to each other via the server, -but it could be slow, especially if the server is located far away. -The enable adhoc direct connections between the FL client and the trainer, -configure the comm_config.json on the client site as follows: - -.. code-block:: json - - { - "allow_adhoc_conns": true, - "use_aio_grpc": true, - "adhoc": { - "scheme": "tcp", - "resources": { - "host": "nvclient", - "secure": true - } - } - } - -This file must be placed into the site's "local" folder within its workspace. - -Pay attention to the following: - -- For most cases, the "scheme" should be set to "tcp" to get the best performance. - If "tcp" cannot be used, you can use "grpc". -- In "resources": +Note that the specified listening_host of a site must be a hostname that +the external trainer can reach via network. - - If FL client and the trainer are within the same trusted network, - you can set "secure" to false; otherwise set it to true. - - The value of the "host" must match the "listening_host" value of the site used in provision. - -Step Three - Prepare Job Configuration --------------------------------------- +Step Two - Prepare Job Configuration +------------------------------------ For each job, configure the config_fed_client.json to use :class:`TaskExchanger` as the executor. -.. code-block:: json +.. code-block:: { "format_version": 2, "executors": [ - { - "tasks": [ - "train" - ], - "executor": { - "path": "nvflare.app_common.executors.task_exchanger.TaskExchanger", - "args": { - "pipe_id": "pipe" - "peer_read_timeout": 30, - "heartbeat_timeout": 60 + { + "tasks": [ + "train" + ], + "executor": { + "path": "nvflare.app_common.executors.task_exchanger.TaskExchanger", + "args": { + "pipe_id": "pipe" + "peer_read_timeout": 30, + "heartbeat_timeout": 60 + } } } - } - ], + ], "task_result_filters": [], "task_data_filters": [], - "components": [] + components = [ + { + id = "pipe" + path = "nvflare.fuel.utils.pipe.cell_pipe.CellPipe" + args { + mode = "PASSIVE" + site_name = "{SITE_NAME}" + token = "{SITE_NAME}" + root_url = "{ROOT_URL}" + secure_mode = "{SECURE_MODE}" + workspace_dir = "{WORKSPACE}" + } + } + ] } Make sure the parameters of the :class:`TaskExchanger` @@ -333,80 +256,95 @@ are configured properly, and change the default values as needed. Please refer to the API page for a detailed explanation of each argument: :class:`TaskExchanger` -Step Four - Trainer Setup -------------------------- +Step Three - Trainer Setup +-------------------------- -The trainer program must have access to a local file system, and you must create a "workspace" folder. -This workspace should be used for all jobs. +For each client site, you will have an FL client and a trainer process. -Copy the "startup" folder of the provisioned site, and put it in the designated workspace folder. -If needed, any additional config files required by the trainer can also be placed in the workspace folder. +To make our integration work, please follow the following steps to +setup the trainer process on each client site: -Ensure to set the FlareAgent's "workspace_dir" to the workspace folder and -that the correct "agent_id" value is passed to both the FL client and the training process. + - Make sure the trainer process has access to a local file system. + - Create a "workspace" folder that is going to be used by this trainer process + This workspace will be used for all jobs. + - Copy the "startup" folder of the client site to this "workspace" folder + If needed, any additional config files required by the trainer can also + be placed in this "workspace" folder. + - Create the trainer script following the steps in the above section. + Please set the FlareAgentWithCellPipe's "workspace_dir" to the path of + this "workspace" folder that you just created. + Please make sure the "agent_id" value of FlareAgentWithCellPipe is the same + as the "token" value in the above Verification ============ -The FL client (TaskExchanger) and your trainer process (FlareAgent) do not have -to be started at exactly the same time. +The FL client (TaskExchanger) and your trainer process (FlareAgentWithCellPipe) +do not have to be started at exactly the same time. + Whichever is started first will wait for the other for ``heartbeat_timeout`` seconds. Once they both are started and connected, you can verify they are directly connected using the Admin console's ``cells`` commands. -The following example shows two clients (red, blue) connected to their external -trainers via the agent_id "ext_trainer_1": +The following example shows two clients (site-1, site-2) connected to their +external trainers via the agent_id/token "ext_trainer": .. code-block:: shell > cells server - server.44c08365-e829-4bc1-a034-cda5a252fe73 - red - red.44c08365-e829-4bc1-a034-cda5a252fe73 - blue - blue.44c08365-e829-4bc1-a034-cda5a252fe73 - red--ndas_1 - blue--ndas_1 - Total Cells: 8 - Done [21695 usecs] 2023-10-16 19:28:37.523651 + server.10d1d3b7-fb50-4c83-9575-e510f32c5d21 + site-1 + site-1.10d1d3b7-fb50-4c83-9575-e510f32c5d21 + site-2 + site-2.10d1d3b7-fb50-4c83-9575-e510f32c5d21 + site-1_ext_trainer_active + site-2_ext_trainer_active + site-2_ext_trainer_passive + site-1_ext_trainer_passive + Total Cells: 10 + The ``cells`` command lists all cells. -Notice that the job 44c08365-e829-4bc1-a034-cda5a252fe73 is running on both "blue" and "red" clients. -Also notice that there are two corresponding ext_trainer cells (red-ext_trainer_1, and blue-ext_trainer1). -.. code-block:: shell +Notice that the job ``10d1d3b7-fb50-4c83-9575-e510f32c5d21`` is running on both +"site-1" and "site-2" clients. - > peers blue--ext_trainer_1 - server - blue.44c08365-e829-4bc1-a034-cda5a252fe73 - Total Agents: 2 - Done [14526 usecs] 2023-10-16 19:28:44.407505 +Also notice that there are two pairs of corresponding cells +(site-1_ext_trainer_active, site-1_ext_trainer_passive) +and ((site-2_ext_trainer_active, site-2_ext_trainer_passive)). -The ``peers`` command shows the cells directly connected to the specified cell. -Here you see that the blue-ext_trainer_1 is directly connected to two cells: -the server and the FL client (blue.44c08365-e829-4bc1-a034-cda5a252fe73). -.. code-block:: shell +Optional - Setup for Adhoc Direct Connection between FL Client and Trainer +========================================================================== + +FL client and the trainer can always talk to each other via the server, +but it could be slow, especially if the server is located far away. +The enable adhoc direct connections between the FL client and the trainer, +configure the comm_config.json on the client site as follows: + +.. code-block:: json - > conns blue--ext_trainer_1 { - "bb_ext_connector": { - "url": "grpc://server:8002", - "handle": "CH00001", - "type": "connector" - }, - "adhoc_connectors": { - "blue.44c08365-e829-4bc1-a034-cda5a252fe73": { - "url": "stcp://nvclient:11947", - "handle": "CH00002", - "type": "connector" + "allow_adhoc_conns": true, + "use_aio_grpc": true, + "adhoc": { + "scheme": "tcp", + "resources": { + "host": "localhost", + "secure": true } } } -The ``conns`` command shows the connectors on the specified cell. -Here you see that blue--ext_trainer_1 has two connectors: -one connects the server on ``grpc://server:8002``, and another connects to -``blue.44c08365-e829-4bc1-a034-cda5a252fe73 on stcp://nvclient:11947``. -Note that this port (11947) is opened by the FL client dynamically. +This file must be placed into the site's "local" folder within its workspace. + +Pay attention to the following: + +- For most cases, the "scheme" should be set to "tcp" to get the best performance. + If "tcp" cannot be used, you can use "grpc". +- In "resources": + + - If FL client and the trainer are within the same trusted network, + you can set "secure" to false; otherwise set it to true. + - The value of the "host" must match the "listening_host" value of the site used in provision. diff --git a/docs/programming_guide/execution_api_type/client_api.rst b/docs/programming_guide/execution_api_type/client_api.rst index 0861a446e3..aff5da50eb 100644 --- a/docs/programming_guide/execution_api_type/client_api.rst +++ b/docs/programming_guide/execution_api_type/client_api.rst @@ -133,7 +133,7 @@ Below is a table overview of key Client APIs. - API Doc Link * - patch - Patches the PyTorch Lightning Trainer for usage with FLARE. - - :func:`train` + - :func:`patch` .. list-table:: Metrics Logger :widths: 25 25 50 diff --git a/docs/release_notes/flare_240.rst b/docs/release_notes/flare_240.rst index d2fe661ad5..4741b386f0 100644 --- a/docs/release_notes/flare_240.rst +++ b/docs/release_notes/flare_240.rst @@ -168,7 +168,7 @@ Improved Job Configuration File Processing - OS Environment Variables - OS environment variables can be referenced via the dollar sign - Parameterized Variable Definition - for creating configuration templates that can be reused and resolved into different concrete configurations -See more details in the :ref:`configuration_files` documentation. +See more details in the :ref:`configurations` documentation. POC Command Upgrade =================== diff --git a/docs/resources/3rd_party_trainer.py b/docs/resources/3rd_party_trainer.py new file mode 100644 index 0000000000..1ffdd085bb --- /dev/null +++ b/docs/resources/3rd_party_trainer.py @@ -0,0 +1,59 @@ +import argparse +import logging + +from nvflare.client.flare_agent import AgentClosed, FlareAgentWithCellPipe + +NUMPY_KEY = "numpy_key" + + +def main(): + + logging.basicConfig() + logging.getLogger().setLevel(logging.INFO) + + parser = argparse.ArgumentParser() + parser.add_argument("--workspace", "-w", type=str, help="workspace folder", required=False, default=".") + parser.add_argument("--site_name", "-s", type=str, help="flare site name", required=True) + parser.add_argument("--agent_id", "-a", type=str, help="agent id", required=True) + + args = parser.parse_args() + + # 1. create the agent + agent = FlareAgentWithCellPipe( + root_url="grpc://server:8002", + site_name=args.site_name, + agent_id=args.agent_id, + workspace_dir=args.workspace, + secure_mode=True, + submit_result_timeout=2.0, + heartbeat_timeout=120.0, + ) + + # 2. start the agent + agent.start() + + # 3. processing tasks + while True: + print("getting task ...") + try: + task = agent.get_task() + except AgentClosed: + print("agent closed - exit") + break + + print(f"got task: {task}") + result = train(task.data) # perform train task + submitted = agent.submit_result(result) + print(f"result submitted: {submitted}") + + # 4. stop the agent + agent.stop() + + +def train(model): + print(f"training on {model}") + return model + + +if __name__ == "__main__": + main() diff --git a/examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_client.json b/examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_client.conf similarity index 100% rename from examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_client.json rename to examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_client.conf diff --git a/examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_server.json b/examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_server.conf similarity index 100% rename from examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_server.json rename to examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-mlflow/app/config/config_fed_server.conf diff --git a/examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/config_fed_client.json b/examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/config_fed_client.conf similarity index 100% rename from examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/config_fed_client.json rename to examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/config_fed_client.conf diff --git a/examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/config_fed_server.json b/examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/config_fed_server.conf similarity index 100% rename from examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/config_fed_server.json rename to examples/advanced/experiment-tracking/mlflow/jobs/hello-pt-tb-mlflow/app/config/config_fed_server.conf diff --git a/examples/hello-world/ml-to-fl/np/README.md b/examples/hello-world/ml-to-fl/np/README.md index c6dea03fc7..1de78e0e5a 100644 --- a/examples/hello-world/ml-to-fl/np/README.md +++ b/examples/hello-world/ml-to-fl/np/README.md @@ -1,4 +1,4 @@ -# Configurations of NVFlare Client API +# NVFlare Client API We will demonstrate how to send back model parameters or model parameters differences in different approaches in the following examples: @@ -18,6 +18,25 @@ We demonstrate how to launch training script once and have training script keeps 1. [Launch once for the whole job](#launch-once-for-the-whole-job) +## Software Requirements + +Please install the requirements first, it is suggested to install inside a virtual environment: + +```bash +pip install -r requirements.txt +``` + +Please also configure the job templates folder: + +```bash +nvflare config -jt ../../../../job_templates/ +nvflare job list_templates +``` + +## Minimum Hardware Requirements + +1 CPU + ## Send model parameters back to the NVFlare server @@ -29,8 +48,6 @@ To send back the whole model parameters, we need to make sure the "params_transf Let reuse the job templates from [sag_np](../../../../job_templates/sag_np/): ```bash -nvflare config -jt ../../../../job_templates/ -nvflare job list_templates nvflare job create -force -j ./jobs/np_param_full_transfer_full -w sag_np -sd ./code/ \ -f config_fed_client.conf app_script=train_full.py params_transfer_type=FULL launch_once=false ``` diff --git a/examples/hello-world/ml-to-fl/pt/README.md b/examples/hello-world/ml-to-fl/pt/README.md index 7789515df8..09c1a8b0db 100644 --- a/examples/hello-world/ml-to-fl/pt/README.md +++ b/examples/hello-world/ml-to-fl/pt/README.md @@ -1,5 +1,19 @@ # PyTorch Deep Learning to Federated Learning transition with NVFlare +We will demonstrate how to transform an existing DL code into an FL application step-by-step: + + 1. [Show a baseline training script](#the-baseline) + 2. [How to modify an existing training script using DL2FL Client API](#transform-cifar10-dl-training-code-to-fl-including-best-model-selection-using-client-api) + 3. [How to modify a structured script using DL2FL decorator](#the-decorator-use-case) + 4. [How to modify a PyTorch Lightning script using DL2FL Lightning Client API](#transform-cifar10-pytorch-lightning-training-code-to-fl-with-nvflare-client-lightning-integration-api) + +If you have multi GPU please refer to the following examples: + + 1. [How to modify a PyTorch DDP training script using DL2FL Client API](#transform-cifar10-pytorch--ddp-training-code-to-fl-using-client-api) + 2. [How to modify a PyTorch Lightning DDP training script using DL2FL Lightning Client API](#transform-cifar10-pytorch-lightning--ddp-training-code-to-fl-with-nvflare-client-lightning-integration-api) + +## Software Requirements + Please install the requirements first, it is suggested to install inside a virtual environment: ```bash @@ -13,17 +27,22 @@ nvflare config -jt ../../../../job_templates/ nvflare job list_templates ``` -We will demonstrate how to transform an existing DL code into an FL application step-by-step: +## Minimum Hardware Requirements - 1. [Show a baseline training script](#the-baseline) - 2. [How to modify an existing training script using DL2FL Client API](#transform-cifar10-dl-training-code-to-fl-including-best-model-selection-using-client-api) - 3. [How to modify a structured script using DL2FL decorator](#the-decorator-use-case) - 4. [How to modify a PyTorch Lightning script using DL2FL Lightning Client API](#transform-cifar10-pytorch-lightning-training-code-to-fl-with-nvflare-client-lightning-integration-api) +Each example has different requirements: -If you have multi GPU please refer to the following examples: +| Example name | minimum requirements | +| ------------ | -------------------- | +| [Show a baseline training script](#the-baseline) | 1 CPU or 1 GPU* | +| [How to modify an existing training script using DL2FL Client API](#transform-cifar10-dl-training-code-to-fl-including-best-model-selection-using-client-api) | 1 CPU or 1 GPU* | +| [How to modify a structured script using DL2FL decorator](#the-decorator-use-case) | 1 CPU or 1 GPU* | +| [How to modify a PyTorch Lightning script using DL2FL Lightning Client API](#transform-cifar10-pytorch-lightning-training-code-to-fl-with-nvflare-client-lightning-integration-api) | 1 CPU or 1 GPU* | +| [How to modify a PyTorch DDP training script using DL2FL Client API](#transform-cifar10-pytorch--ddp-training-code-to-fl-using-client-api) | 2 GPUs | +| [How to modify a PyTorch Lightning DDP training script using DL2FL Lightning Client API](#transform-cifar10-pytorch-lightning--ddp-training-code-to-fl-with-nvflare-client-lightning-integration-api) | 2 CPUs or 2 GPUs** | - 1. [How to modify a PyTorch DDP training script using DL2FL Client API](#transform-cifar10-pytorch--ddp-training-code-to-fl-using-client-api) - 2. [How to modify a PyTorch Lightning DDP training script using DL2FL Lightning Client API](#transform-cifar10-pytorch-lightning--ddp-training-code-to-fl-with-nvflare-client-lightning-integration-api) + +\* it depends on you use `device=cpu` or `device=cuda` +\*\* it depends on whether `torch.cuda.is_available()` is True or not ## The baseline @@ -200,8 +219,6 @@ nvflare simulator -n 2 -t 2 ./jobs/lightning -w lightning_workspace We follow the official [PyTorch documentation](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#initialize-ddp-with-torch-distributed-run-torchrun) and write a [./code/cifar10_ddp_original.py](./code/cifar10_ddp_original.py). -Note that this example requires at least 2 GPUs on your machine. - Note that we wrap the evaluation logic into a method for better usability. It can be run using the torch distributed run: diff --git a/examples/hello-world/ml-to-fl/pt/code/cifar10_lightning_ddp_fl.py b/examples/hello-world/ml-to-fl/pt/code/cifar10_lightning_ddp_fl.py index 62a513f548..0398c9d3b0 100644 --- a/examples/hello-world/ml-to-fl/pt/code/cifar10_lightning_ddp_fl.py +++ b/examples/hello-world/ml-to-fl/pt/code/cifar10_lightning_ddp_fl.py @@ -72,7 +72,9 @@ def main(): model = LitNet() cifar10_dm = CIFAR10DataModule() - trainer = Trainer(max_epochs=1, strategy="ddp", devices=2 if torch.cuda.is_available() else None) + trainer = Trainer( + max_epochs=1, strategy="ddp", devices=2, accelerator="gpu" if torch.cuda.is_available() else "cpu" + ) # (2) patch the lightning trainer flare.patch(trainer) diff --git a/examples/hello-world/ml-to-fl/tf/README.md b/examples/hello-world/ml-to-fl/tf/README.md index 80b879425b..e2444844f4 100644 --- a/examples/hello-world/ml-to-fl/tf/README.md +++ b/examples/hello-world/ml-to-fl/tf/README.md @@ -1,18 +1,38 @@ # TensorFlow Deep Learning to Federated Learning transition with NVFlare +We will demonstrate how to transform an existing DL code into an FL application step-by-step: + +1. [How to modify an existing training script using DL2FL Client API](#transform-cifar10-tensorflow-training-code-to-fl-with-nvflare-client-api) + +2. [How to modify an existing multi GPU training script using DL2FL Client API](#transform-cifar10-tensorflow-multi-gpu-training-code-to-fl-with-nvflare-client-api) + +## Software Requirements + Please install the requirements first, it is suggested to install inside a virtual environment: ```bash pip install -r requirements.txt ``` -Note that for running with GPUs, we recommend using [NVIDIA TensorFlow docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) +Please also configure the job templates folder: -We will demonstrate how to transform an existing DL code into an FL application step-by-step: +```bash +nvflare config -jt ../../../../job_templates/ +nvflare job list_templates +``` -1. [How to modify an existing training script using DL2FL Client API](#transform-cifar10-tensorflow-training-code-to-fl-with-nvflare-client-api) +## Minimum Hardware Requirements + +| Example name | minimum requirements | +| ------------ | -------------------- | +| [How to modify an existing training script using DL2FL Client API](#transform-cifar10-tensorflow-training-code-to-fl-with-nvflare-client-api) | 1 CPU or 1 GPU* | +| [How to modify an existing multi GPU training script using DL2FL Client API](#transform-cifar10-tensorflow-multi-gpu-training-code-to-fl-with-nvflare-client-api) | 2 CPUs or 2 GPUs* | + +\* depends on whether TF can found GPU or not + + +Note that for running with GPUs, we recommend using [NVIDIA TensorFlow docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) -2. [How to modify an existing multi GPU training script using DL2FL Client API](#transform-cifar10-tensorflow-multi-gpu-training-code-to-fl-with-nvflare-client-api) ## Transform CIFAR10 TensorFlow training code to FL with NVFLARE Client API @@ -46,7 +66,6 @@ Please refer to [JOB CLI tutorial](../../../tutorials/job_cli.ipynb) on how to g We choose the [tensorflow job template](../../../../job_templates/sag_tf/) and run the following command to create the job: ```bash -nvflare config -jt ../../../../job_templates nvflare job create -force -j ./jobs/tensorflow -w sag_tf -sd ./code/ -f config_fed_client.conf app_script=cifar10_tf_fl.py ``` @@ -82,7 +101,6 @@ Please refer to [JOB CLI tutorial](../../../tutorials/job_cli.ipynb) on how to g We choose the [tensorflow job template](../../../../job_templates/sag_tf/) and run the following command to create the job: ```bash -nvflare config -jt ../../../../job_templates nvflare job create -force -j ./jobs/tensorflow_multi_gpu -w sag_tf -sd ./code/ -f config_fed_client.conf app_script=cifar10_tf_multi_gpu_fl.py ``` diff --git a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_client.json b/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_client.conf similarity index 100% rename from integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_client.json rename to integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_client.conf diff --git a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.json b/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.conf similarity index 95% rename from integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.json rename to integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.conf index 749b59deb5..43650a1a9d 100644 --- a/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.json +++ b/integration/monai/examples/spleen_ct_segmentation_local/jobs/spleen_ct_segmentation_local/app/config/config_fed_server.conf @@ -52,10 +52,10 @@ "experiment_name": "monai-spleen-experiment", "run_name": "monai-spleen-with-mlflow", "experiment_tags": { - "mlflow-note-content": "## **MONAI experiment with spleen bundle with MLflow**" + "mlflow.note.content": "## **MONAI experiment with spleen bundle with MLflow**" }, "run_tags": { - "mlflow-note-content": "## Federated Experiment tracking with MLflow \n### Example of using **[NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html)** to train and run MONAI-bundle using federated averaging ([FedAvg]([FedAvg](https://arxiv.org/abs/1602.05629))) and [PyTorch](https://pytorch.org/) as the deep learning training framework. This example also highlights the FLARE streaming capability from the clients to the server for server delivery to MLflow.\n" + "mlflow.note.content": "## Federated Experiment tracking with MLflow \n### Example of using **[NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html)** to train and run MONAI-bundle using federated averaging ([FedAvg]([FedAvg](https://arxiv.org/abs/1602.05629))) and [PyTorch](https://pytorch.org/) as the deep learning training framework. This example also highlights the FLARE streaming capability from the clients to the server for server delivery to MLflow.\n" } }, "artifact_location": "artifacts" diff --git a/nvflare/client/lightning/__init__.py b/nvflare/client/lightning/__init__.py index a3f1d5acbb..395e6728ab 100644 --- a/nvflare/client/lightning/__init__.py +++ b/nvflare/client/lightning/__init__.py @@ -12,6 +12,16 @@ # See the License for the specific language governing permissions and # limitations under the License. +"""PyTorch Lightning API integration module for simplified imports. + +Usage: + from nvflare.client.lightning import patch + +For detailed information on usage and the API, refer to: + :mod:`nvflare.app_opt.lightning.api` + +""" + from nvflare.fuel.utils.import_utils import optional_import pytorch_lightning, ok = optional_import(module="pytorch_lightning") diff --git a/nvflare/lighter/dummy_project.yml b/nvflare/lighter/dummy_project.yml index fb5a759b95..57311da4ae 100644 --- a/nvflare/lighter/dummy_project.yml +++ b/nvflare/lighter/dummy_project.yml @@ -12,9 +12,10 @@ participants: - name: site-1 type: client org: nvidia - # listening_host will enable creating one pair of cert/private key for this client - # so it can behave like a server for client api. The value must be a hostname that - # client api can reach via network. + # Specifying listening_host will enable the creation of one pair of + # certificate/private key for this client, allowing the client to function + # as a server for 3rd-party integration. + # The value must be a hostname that the external trainer can reach via the network. # listening_host: site-1-lh - name: site-2 type: client diff --git a/setup.cfg b/setup.cfg index 23868dbe05..7b1b6e834f 100644 --- a/setup.cfg +++ b/setup.cfg @@ -46,6 +46,8 @@ PT = SKLEARN = scikit-learn TRACKING = + mlflow + wandb tensorboard CONFIG = omegaconf @@ -55,6 +57,8 @@ app_opt = %(PT)s %(SKLEARN)s %(TRACKING)s + pytorch_lightning + xgboost app_opt_mac = %(PT)s %(SKLEARN)s From 78e493b5d0bfc4de4108ed386464fe6808d97375 Mon Sep 17 00:00:00 2001 From: Holger Roth <6304754+holgerroth@users.noreply.github.com> Date: Fri, 2 Feb 2024 12:07:32 -0500 Subject: [PATCH 27/39] fix plot script (#2346) --- .../cifar10/cifar10-sim/figs/plot_tensorboard_events.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/advanced/cifar10/cifar10-sim/figs/plot_tensorboard_events.py b/examples/advanced/cifar10/cifar10-sim/figs/plot_tensorboard_events.py index 0e3c06a468..40284f3596 100644 --- a/examples/advanced/cifar10/cifar10-sim/figs/plot_tensorboard_events.py +++ b/examples/advanced/cifar10/cifar10-sim/figs/plot_tensorboard_events.py @@ -93,7 +93,7 @@ def main(): for config, exp in experiments.items(): config_name = config.split(" ")[0] alpha = exp.get("alpha", None) - if alpha: + if alpha is not None: config_name = config_name + f"*alpha{alpha}" else: raise ValueError(f"Expected an alpha value to be provided but got alpha={alpha}") From 5fae920b3b4becc2f8a1f300300f2d8c7b2e2640 Mon Sep 17 00:00:00 2001 From: Zhihong Zhang <100308595+nvidianz@users.noreply.github.com> Date: Fri, 2 Feb 2024 12:18:16 -0500 Subject: [PATCH 28/39] Added a few workarounds for HTTP driver's latency issues (#2343) Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> --- nvflare/fuel/f3/drivers/aio_http_driver.py | 13 ++++++++++--- nvflare/fuel/f3/streaming/byte_receiver.py | 2 +- nvflare/fuel/f3/streaming/byte_streamer.py | 2 +- nvflare/private/fed/client/fed_client_base.py | 2 +- 4 files changed, 13 insertions(+), 6 deletions(-) diff --git a/nvflare/fuel/f3/drivers/aio_http_driver.py b/nvflare/fuel/f3/drivers/aio_http_driver.py index f8e0f95d5a..61c953a867 100644 --- a/nvflare/fuel/f3/drivers/aio_http_driver.py +++ b/nvflare/fuel/f3/drivers/aio_http_driver.py @@ -85,7 +85,8 @@ async def _async_send_frame(self, frame: BytesAlike): # This is to yield control. See bug: https://github.com/aaugustin/websockets/issues/865 await asyncio.sleep(0) except Exception as ex: - log.error(f"Error sending frame for connection {self}: {secure_format_exception(ex)}") + log.error(f"Error sending frame for connection {self}, closing: {secure_format_exception(ex)}") + self.close() class AioHttpDriver(BaseDriver): @@ -184,5 +185,11 @@ async def _handler(self, websocket): async def _read_loop(conn: WsConnection): while not conn.closing: # Reading from websocket and call receiver CB - frame = await conn.websocket.recv() - conn.process_frame(frame) + try: + frame = await conn.websocket.recv() + conn.process_frame(frame) + except ConnectionClosedOK as ex: + raise ex + except Exception as ex: + log.error(f"Exception {type(ex)} on connection {conn}: {ex}") + raise ex diff --git a/nvflare/fuel/f3/streaming/byte_receiver.py b/nvflare/fuel/f3/streaming/byte_receiver.py index 96aa9f2ff5..21244ab4aa 100644 --- a/nvflare/fuel/f3/streaming/byte_receiver.py +++ b/nvflare/fuel/f3/streaming/byte_receiver.py @@ -39,7 +39,7 @@ MAX_OUT_SEQ_CHUNKS = 16 # 1/4 of the window size ACK_INTERVAL = 1024 * 1024 * 4 -READ_TIMEOUT = 60 +READ_TIMEOUT = 300 COUNTER_NAME_RECEIVED = "received" diff --git a/nvflare/fuel/f3/streaming/byte_streamer.py b/nvflare/fuel/f3/streaming/byte_streamer.py index e06fbed2b9..437a292c14 100644 --- a/nvflare/fuel/f3/streaming/byte_streamer.py +++ b/nvflare/fuel/f3/streaming/byte_streamer.py @@ -38,7 +38,7 @@ STREAM_CHUNK_SIZE = 1024 * 1024 STREAM_WINDOW_SIZE = 16 * STREAM_CHUNK_SIZE -STREAM_ACK_WAIT = 10 +STREAM_ACK_WAIT = 60 STREAM_TYPE_BYTE = "byte" STREAM_TYPE_BLOB = "blob" diff --git a/nvflare/private/fed/client/fed_client_base.py b/nvflare/private/fed/client/fed_client_base.py index 6abd8068f9..b6c01f8d43 100644 --- a/nvflare/private/fed/client/fed_client_base.py +++ b/nvflare/private/fed/client/fed_client_base.py @@ -93,7 +93,7 @@ def __init__( cell=cell, client_register_interval=client_args.get("client_register_interval", 2.0), timeout=client_args.get("communication_timeout", 30.0), - maint_msg_timeout=client_args.get("maint_msg_timeout", 5.0), + maint_msg_timeout=client_args.get("maint_msg_timeout", 30.0), ) self.secure_train = secure_train From 8c10f567e8b7469e9e06aedafc96cd79b7e5a6ce Mon Sep 17 00:00:00 2001 From: Sean Yang Date: Fri, 2 Feb 2024 09:38:04 -0800 Subject: [PATCH 29/39] Address VDR report (#2335) * address vdr report * address comments * update simulator to processes --------- Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> --- docs/flare_overview.rst | 2 +- docs/getting_started.rst | 13 ++++++-- docs/index.rst | 11 ++++--- .../programming_guide/experiment_tracking.rst | 25 ++++++++++---- docs/user_guide/nvflare_cli/fl_simulator.rst | 33 +++++++++++-------- .../experiment-tracking/wandb/README.md | 4 ++- examples/hello-world/step-by-step/README.md | 2 +- .../step-by-step/cifar10/cse/cse.ipynb | 4 +-- .../step-by-step/cifar10/cyclic/cyclic.ipynb | 5 ++- .../cifar10/cyclic_ccwf/cyclic_ccwf.ipynb | 4 ++- .../step-by-step/cifar10/sag/sag.ipynb | 5 ++- .../sag_deploy_map/sag_deploy_map.ipynb | 5 ++- .../cifar10/sag_executor/sag_executor.ipynb | 7 +++- .../step-by-step/cifar10/sag_he/sag_he.ipynb | 5 ++- .../cifar10/sag_mlflow/sag_mlflow.ipynb | 10 +++--- .../sag_model_learner/sag_model_learner.ipynb | 4 ++- .../cifar10/stats/image_stats.ipynb | 5 ++- .../step-by-step/cifar10/swarm/swarm.ipynb | 5 ++- .../hello-world/step-by-step/higgs/README.md | 3 +- .../higgs/sklearn-kmeans/sklearn_kmeans.ipynb | 5 ++- .../higgs/sklearn-linear/sklearn_linear.ipynb | 6 ++-- .../higgs/sklearn-svm/sklearn_svm.ipynb | 4 ++- .../higgs/stats/tabular_stats.ipynb | 7 ++-- .../higgs/xgboost/xgboost_horizontal.ipynb | 6 +++- examples/tutorials/job_cli.ipynb | 4 ++- integration/monai/README.md | 4 --- integration/nemo/examples/peft/README.md | 2 ++ .../nemo/examples/prompt_learning/README.md | 2 ++ .../prompt_learning/prompt_learning.ipynb | 8 ++++- .../examples/supervised_fine_tuning/README.md | 1 + job_templates/readme.md | 4 ++- 31 files changed, 143 insertions(+), 62 deletions(-) diff --git a/docs/flare_overview.rst b/docs/flare_overview.rst index c2f6ecfb91..15eaafa8d3 100644 --- a/docs/flare_overview.rst +++ b/docs/flare_overview.rst @@ -26,7 +26,7 @@ Built for productivity FLARE is designed for maximum productivity, providing a range of tools to enhance user experience and research efficiency at different stages of the development process: - **FLARE Client API:** Enables users to transition seamlessly from ML/DL to FL with just a few lines of code changes. -- **Simulator CLI:** Allows users to simulate federated learning or computing jobs in multi-thread settings within a single computer, offering quick response and debugging. The same job can be deployed directly to production. +- **Simulator CLI:** Allows users to simulate federated learning or computing jobs in multi-process settings within a single computer, offering quick response and debugging. The same job can be deployed directly to production. - **POC CLI:** Facilitates the simulation of federated learning or computing jobs in multi-process settings within one computer. Different processes represent server, clients, and an admin console, providing users with a realistic sense of the federated network. It also allows users to simulate project deployment on a single host. - **Job CLI:** Permits users to create and submit jobs directly in POC or production environments. - **FLARE API:** Enables users to run jobs directly from Python code or notebooks. diff --git a/docs/getting_started.rst b/docs/getting_started.rst index 6f896b17b4..cee26e8f9e 100644 --- a/docs/getting_started.rst +++ b/docs/getting_started.rst @@ -66,6 +66,11 @@ Installation .. note:: The server and client versions of nvflare must match, we do not support cross-version compatibility. +Supported Operating Systems +--------------------------- +- Linux +- OSX (Note: some optional dependencies are not compatible, such as tenseal and openmined.psi) + Python Version -------------- @@ -120,7 +125,6 @@ You may find that the pip and setuptools versions in the venv need updating: (nvflare-env) $ python3 -m pip install -U pip (nvflare-env) $ python3 -m pip install -U setuptools - Install Stable Release ---------------------- @@ -130,6 +134,11 @@ Stable releases are available on `NVIDIA FLARE PyPI ` for modules and components with optional dependencies. .. _containerized_deployment: @@ -213,7 +222,7 @@ Production mode is secure with TLS certificates - depending the choice the deplo - HA or non-HA - Local or remote - - On-premise or on cloud + - On-premise or on cloud (See :ref:`cloud_deployment`) Using non-HA, secure, local mode (all clients and server running on the same host), production mode is very similar to POC mode except it is secure. diff --git a/docs/index.rst b/docs/index.rst index 3e73c525fd..1c8527e62c 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -48,18 +48,21 @@ and simulation to real-world production deployment. Some of the key components - **Management tools** for secure provisioning and deployment, orchestration, and management - **Specification-based API** for extensibility -Learn more in the :ref:`FLARE Overview `, :ref:`What's New `, and the -:ref:`User Guide ` and :ref:`Programming Guide `. +Learn more about FLARE features in the :ref:`FLARE Overview ` and :ref:`What's New `. Getting Started =============== For first-time users and FL researchers, FLARE provides the :ref:`FL Simulator ` that allows you to build, test, and deploy applications locally. The :ref:`Getting Started ` guide covers installation and walks through an example application using the FL Simulator. +Additional examples can be found at the :ref:`Examples Applications `, which showcase different federated learning workflows and algorithms on various machine learning and deep learning tasks. +FLARE for Users +=============== +If you want to learn how to interact with the FLARE system, please refer to the :ref:`User Guide `. When you are ready to for a secure, distributed deployment, the :ref:`Real World Federated Learning ` section covers the tools and process required to deploy and operate a secure, real-world FLARE project. FLARE for Developers ==================== -When you're ready to build your own application, the :ref:`Programming Best Practices `, :ref:`FAQ`, and -:ref:`Programming Guide ` give an in depth look at the FLARE platform and APIs. +When you're ready to build your own application, the :ref:`Programming Guide `, :ref:`Programming Best Practices `, :ref:`FAQ`, and :ref:`API Reference ` +give an in depth look at the FLARE platform and APIs. diff --git a/docs/programming_guide/experiment_tracking.rst b/docs/programming_guide/experiment_tracking.rst index 06a274ee81..c0ccbbe939 100644 --- a/docs/programming_guide/experiment_tracking.rst +++ b/docs/programming_guide/experiment_tracking.rst @@ -37,6 +37,13 @@ provided examples, the Receiver is on the FL server, but it could also be on the - Server-side experiment tracking also can organize different clients' results into different experiment runs so they can be easily compared side-by-side. +.. note:: + + This page covers experiment tracking using :class:`LogWriters `, + which are configured and used with :ref:`executor` or :ref:`model_learner` on the FLARE-side code. + However if using the Client API, please refer to :ref:`client_api` and :ref:`nvflare.client.tracking` for adding experiment tracking to your custom training code. + + ************************************** Tools, Sender, LogWriter and Receivers ************************************** @@ -60,9 +67,9 @@ where the actual experiment logs are recorded. The components that receive these logs are called Receivers based on :class:`AnalyticsReceiver `. The receiver component leverages the experiment tracking tool and records the logs during the experiment run. -In a normal setting, we would have pairs of sender and receivers, such as: +In a normal setting, we would have pairs of sender and receivers, with some provided implementations in :mod:`nvflare.app_opt.tracking`: - - TBWriter <-> TBReceiver + - TBWriter <-> TBAnalyticsReceiver - MLflowWriter <-> MLflowReceiver - WandBWriter <-> WandBReceiver @@ -94,13 +101,11 @@ There are three things to consider for developing a custom experiment tracking t Data Type ========= -Currently, the supported data types are metrics, params, and text. If you require other data types, may sure you add -the type to :class:`AnalyticsDataType `. +Currently, the supported data types are listed in :class:`AnalyticsDataType `, and other data types can be added as needed. Writer ====== - -Implement LogWriter interface with the API syntax. For each tool, we mimic the API syntax of the underlying tool, +Implement :class:`LogWriter ` interface with the API syntax. For each tool, we mimic the API syntax of the underlying tool, so users can use what they are familiar with without learning a new API. For example, for Tensorboard, TBWriter uses add_scalar() and add_scalars(); for MLflow, the syntax is log_metric(), log_metrics(), log_parameter(), and log_parameters(); for W&B, the writer just has log(). @@ -109,7 +114,7 @@ The data collected with these calls will all send to the AnalyticsSender to deli Receiver ======== -Implement AnalyticsReceiver interface and determine how to represent different sites' logs. In all three implementations +Implement :class:`AnalyticsReceiver ` interface and determine how to represent different sites' logs. In all three implementations (Tensorboard, MLflow, WandB), each site's log is represented as one run. Depending on the individual tool, the implementation can be different. For example, for both Tensorboard and MLflow, we create different runs for each client and map to the site name. In the WandB implementation, we have to leverage multiprocess and let each run in a different process. @@ -121,13 +126,19 @@ Examples Overview The :github_nvflare_link:`experiment tracking examples ` illustrate how to leverage different writers and receivers. All examples are based upon the hello-pt example. +TensorBoard +=========== The example in the "tensorboard" directory shows how to use the Tensorboard Tracking Tool (for both the sender and receiver). See :ref:`tensorboard_streaming` for details. +MLflow +====== Under the "mlflow" directory, the "hello-pt-mlflow" job shows how to use MLflow for tracking with both the MLflow sender and receiver. The "hello-pt-tb-mlflow" job shows how to use the Tensorboard Sender, while the receiver is MLflow. See :ref:`experiment_tracking_mlflow` for details. +Weights & Biases +================ Under the :github_nvflare_link:`wandb ` directory, the "hello-pt-wandb" job shows how to use Weights and Biases for experiment tracking with the WandBWriter and WandBReceiver to log metrics. diff --git a/docs/user_guide/nvflare_cli/fl_simulator.rst b/docs/user_guide/nvflare_cli/fl_simulator.rst index b0ffe8fa10..f52fd47078 100644 --- a/docs/user_guide/nvflare_cli/fl_simulator.rst +++ b/docs/user_guide/nvflare_cli/fl_simulator.rst @@ -49,7 +49,7 @@ Command examples Run a single NVFlare app ======================== -This command will run the same ``hello-numpy-sag`` app on the server and 8 clients using 1 thread. The client names will be site-1, site-2, ... , site-8: +This command will run the same ``hello-numpy-sag`` app on the server and 8 clients using 1 process. The client names will be site-1, site-2, ... , site-8: .. code-block:: python @@ -829,22 +829,29 @@ application run. status = run_simulator(args) sys.exit(status) -**************************** -Threads, Clients, and Events -**************************** +****************************** +Processes, Clients, and Events +****************************** -Specifying threads -================== -The simulator ``-t`` option provides the ability to specify how many threads to run the simulator with. +Specifying number of processes +============================== +The simulator ``-t`` option provides the ability to specify how many processes to run the simulator with. -When you run the simulator with ``-t 1``, there is only one client active and running at a time, and the clients will be running in -turn. This is to enable the simulation of large number of clients using a single machine with limited resources. +.. note:: + + The ``-t`` and ``--threads`` option for simulator was originally due to clients running in separate threads. + However each client now actually runs in a separate process. This distinction will not affect the user experience. + +- N = number of clients (``-n``) +- T = number of processes (``-t``) -Note that if you have fewer threads than the number of clients, ClientRunner/learner object will go thorugh setup and -teardown in every round. +When running the simulator with fewer processes than clients (T < N) +the simulator will need to swap-in/out the clients for the processes, resulting in some of the clients running sequentially as processes are available. +This also will cause the ClientRunner/learner objects to go through setup and teardown in every round. +Using T < N is only needed when trying to simulate of large number of clients using a single machine with limited resources. -With ``-t=num_client``, the simulator will run the number of clients in separate threads at the same time. Each -client will always be running in memory with no swap_in / swap_out, but it will require more resources available. +In most cases, run the simulator with the same number of processes as clients (T = N). The simulator will run the number of clients in separate processes at the same time. Each +client will always be running in memory with no swap-in/out, but it will require more resources available. For the dataset / tensorboard initialization, you could make use of EventType.SWAP_IN and EventType.SWAP_OUT in the application. diff --git a/examples/advanced/experiment-tracking/wandb/README.md b/examples/advanced/experiment-tracking/wandb/README.md index 27d06fa243..aaadc13b55 100644 --- a/examples/advanced/experiment-tracking/wandb/README.md +++ b/examples/advanced/experiment-tracking/wandb/README.md @@ -26,7 +26,9 @@ export PYTHONPATH=${PWD}/.. Import the W&B Python SDK and log in: ``` -wandb.login() +python3 +>>> import wandb +>>> wandb.login() ``` Provide your API key when prompted. diff --git a/examples/hello-world/step-by-step/README.md b/examples/hello-world/step-by-step/README.md index aba2957283..59a9c487b0 100644 --- a/examples/hello-world/step-by-step/README.md +++ b/examples/hello-world/step-by-step/README.md @@ -7,7 +7,7 @@ To run the notebooks in each example, please make sure you first set up a virtua These step-by-step example series are aimed to help users quickly get started and learn about FLARE. For consistency, each example in the series uses the same dataset- CIFAR10 for image data and the HIGGS dataset for tabular data. -The examples will build upon previous ones to showcase different features, workflows, or APIs, allowing users to gain a comprehensive understanding of FLARE functionalities. See the README in each directory for more details about each series. +The examples will build upon previous ones to showcase different features, workflows, or APIs, allowing users to gain a comprehensive understanding of FLARE functionalities (Note: each example is self-contained, so going through them in order is not required, but recommended). See the README in each directory for more details about each series. ## Common Questions diff --git a/examples/hello-world/step-by-step/cifar10/cse/cse.ipynb b/examples/hello-world/step-by-step/cifar10/cse/cse.ipynb index 176fc875f7..a25c7ffd32 100644 --- a/examples/hello-world/step-by-step/cifar10/cse/cse.ipynb +++ b/examples/hello-world/step-by-step/cifar10/cse/cse.ipynb @@ -180,9 +180,9 @@ "id": "48271064", "metadata": {}, "source": [ - "For additional resources, see other examples for SAG with CSE using the [ModelLearner](../sag_model_learner/sag_model_learner.ipynb), [Executor](../sag_executor/sag_executor.ipynb), and [Hello-Numpy](https://github.com/NVIDIA/NVFlare/tree/main/examples/hello-world/hello-numpy-cross-val).\n", + "For additional resources, see other examples for SAG with CSE using the [ModelLearner](../sag_model_learner/sag_model_learner.ipynb) and [Executor](../sag_executor/sag_executor.ipynb). [Hello-Numpy](https://github.com/NVIDIA/NVFlare/tree/main/examples/hello-world/hello-numpy-cross-val) also demonstrates how to run cross-site evaluation using the previous training results.\n", "\n", - "Also the ability to run Cross-site Evaluation without having to re-run training will be added in the near future." + "Next we will look at the [cyclic](../cyclic/cyclic.ipynb) example, which shows the cyclic workflow for the Cyclic Weight Transfer algorithm." ] }, { diff --git a/examples/hello-world/step-by-step/cifar10/cyclic/cyclic.ipynb b/examples/hello-world/step-by-step/cifar10/cyclic/cyclic.ipynb index 153a4b6468..5dbf03dd2c 100644 --- a/examples/hello-world/step-by-step/cifar10/cyclic/cyclic.ipynb +++ b/examples/hello-world/step-by-step/cifar10/cyclic/cyclic.ipynb @@ -140,7 +140,10 @@ "id": "48271064", "metadata": {}, "source": [ - "As an additional resource, also see the [hello-cyclic](../../../../hello-world/hello-cyclic/README.md) for a Tensorflow Executor implementation using the MNIST dataset." + "As an additional resource, also see the [hello-cyclic](../../../../hello-world/hello-cyclic/README.md) for a Tensorflow Executor implementation using the MNIST dataset.\n", + "\n", + "While this example focused on the server-controlled cyclic workflow, now we will introduce the idea of client-controlled workflows.\n", + "The next [cyclic_ccwf](../cyclic_ccwf/cyclic_ccwf.ipynb) example is a client-controlled version of the cyclic workflow." ] }, { diff --git a/examples/hello-world/step-by-step/cifar10/cyclic_ccwf/cyclic_ccwf.ipynb b/examples/hello-world/step-by-step/cifar10/cyclic_ccwf/cyclic_ccwf.ipynb index f90ce77d13..778943998e 100644 --- a/examples/hello-world/step-by-step/cifar10/cyclic_ccwf/cyclic_ccwf.ipynb +++ b/examples/hello-world/step-by-step/cifar10/cyclic_ccwf/cyclic_ccwf.ipynb @@ -145,7 +145,9 @@ "cell_type": "markdown", "id": "9bef3134", "metadata": {}, - "source": [] + "source": [ + "Lastly, we have the [swarm](../swarm/swarm.ipynb) example, which covers swarm learning and client-controlled cross-site evaluation workflows." + ] } ], "metadata": { diff --git a/examples/hello-world/step-by-step/cifar10/sag/sag.ipynb b/examples/hello-world/step-by-step/cifar10/sag/sag.ipynb index 3ef045687f..f29f41249e 100644 --- a/examples/hello-world/step-by-step/cifar10/sag/sag.ipynb +++ b/examples/hello-world/step-by-step/cifar10/sag/sag.ipynb @@ -262,7 +262,10 @@ "id": "b055bde7-432d-4e6b-9163-b5ab7ede7b73", "metadata": {}, "source": [ - "The job should be running in the simulator mode. We are done with the training. " + "The job should be running in the simulator mode. We are done with the training. \n", + "\n", + "The next 5 examples will use the same ScatterAndGather workflow, but will demonstrate different execution APIs and feature.\n", + "In the next example [sag_deploy_map](../sag_deploy_map/sag_deploy_map.ipynb), we will learn about the deploy_map configuration for deployment of apps to different sites." ] } ], diff --git a/examples/hello-world/step-by-step/cifar10/sag_deploy_map/sag_deploy_map.ipynb b/examples/hello-world/step-by-step/cifar10/sag_deploy_map/sag_deploy_map.ipynb index 91d4beef0d..b59dbad293 100644 --- a/examples/hello-world/step-by-step/cifar10/sag_deploy_map/sag_deploy_map.ipynb +++ b/examples/hello-world/step-by-step/cifar10/sag_deploy_map/sag_deploy_map.ipynb @@ -261,7 +261,10 @@ "id": "0af8036f-1f94-426d-8eb7-6e8b9be70a7e", "metadata": {}, "source": [ - "The job should be running in the simulator mode. We are done with the training. " + "The job should be running in the simulator mode. We are done with the training. \n", + "\n", + "In the next example [sag_model_learner](../sag_model_learner/sag_model_learner.ipynb), we will illustrate how to use the Model Learner API instead of the Client API,\n", + "and highlight why and when to use it." ] } ], diff --git a/examples/hello-world/step-by-step/cifar10/sag_executor/sag_executor.ipynb b/examples/hello-world/step-by-step/cifar10/sag_executor/sag_executor.ipynb index aed5f6ba74..e71dd87cbf 100644 --- a/examples/hello-world/step-by-step/cifar10/sag_executor/sag_executor.ipynb +++ b/examples/hello-world/step-by-step/cifar10/sag_executor/sag_executor.ipynb @@ -222,7 +222,12 @@ "id": "48271064", "metadata": {}, "source": [ - "For additional resources, take a look at the various other executors with different use cases in the app_common, app_opt, and examples folder." + "For additional resources, take a look at the various other executors with different use cases in the app_common, app_opt, and examples folder.\n", + "\n", + "In the previous examples we have finished covering each of Execution API types: the Client API, Model Learner, and Executor.\n", + "Now we will be using the Client API in future examples to highlight other features and workflows.\n", + "\n", + "Next we have the [sag_mlflow](../sag_mlflow/sag_mlflow.ipynb) example, which shows how to enable MLflow experiment tracking logs." ] }, { diff --git a/examples/hello-world/step-by-step/cifar10/sag_he/sag_he.ipynb b/examples/hello-world/step-by-step/cifar10/sag_he/sag_he.ipynb index 12936dd208..c80f7d37af 100644 --- a/examples/hello-world/step-by-step/cifar10/sag_he/sag_he.ipynb +++ b/examples/hello-world/step-by-step/cifar10/sag_he/sag_he.ipynb @@ -197,7 +197,10 @@ "id": "b19da336", "metadata": {}, "source": [ - "As an additional resource, see the [CIFAR10 Real World Example](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/cifar10/cifar10-real-world) for creating a secure workspace for HE using provisioning instead of POC mode." + "As an additional resource, see the [CIFAR10 Real World Example](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/cifar10/cifar10-real-world) for creating a secure workspace for HE using provisioning instead of POC mode.\n", + "\n", + "Now we will begin to take a look at other workflows besides ScatterAndGather.\n", + "First we have the [cse](../cse/cse.ipynb) example, which shows the server-controlled cross-site evaluation workflow." ] } ], diff --git a/examples/hello-world/step-by-step/cifar10/sag_mlflow/sag_mlflow.ipynb b/examples/hello-world/step-by-step/cifar10/sag_mlflow/sag_mlflow.ipynb index cb39afaf61..fa295c0e3b 100644 --- a/examples/hello-world/step-by-step/cifar10/sag_mlflow/sag_mlflow.ipynb +++ b/examples/hello-world/step-by-step/cifar10/sag_mlflow/sag_mlflow.ipynb @@ -183,12 +183,12 @@ ] }, { - "cell_type": "code", - "execution_count": null, - "id": "e69c9ed2-359a-4f97-820f-25e9323a4e92", + "cell_type": "markdown", + "id": "58037d1e", "metadata": {}, - "outputs": [], - "source": [] + "source": [ + "Next we will look at the [sag_he](../sag_he/sag_he.ipynb) example, which demonstrates how to enable homomorphic encryption using the POC -he mode." + ] } ], "metadata": { diff --git a/examples/hello-world/step-by-step/cifar10/sag_model_learner/sag_model_learner.ipynb b/examples/hello-world/step-by-step/cifar10/sag_model_learner/sag_model_learner.ipynb index 261275427c..ae8aec8a6c 100644 --- a/examples/hello-world/step-by-step/cifar10/sag_model_learner/sag_model_learner.ipynb +++ b/examples/hello-world/step-by-step/cifar10/sag_model_learner/sag_model_learner.ipynb @@ -204,7 +204,9 @@ "id": "48271064", "metadata": {}, "source": [ - "As an additional resource, also see the [CIFAR10 examples](../../../../advanced/cifar10/README.md) for a comprehensive implementation of a PyTorch ModelLearner." + "As an additional resource, also see the [CIFAR10 examples](../../../../advanced/cifar10/README.md) for a comprehensive implementation of a PyTorch ModelLearner.\n", + "\n", + "In the next example [sag_executor](../sag_executor/sag_executor.ipynb), we will illustrate how to use the Executor API for more specific use cases." ] }, { diff --git a/examples/hello-world/step-by-step/cifar10/stats/image_stats.ipynb b/examples/hello-world/step-by-step/cifar10/stats/image_stats.ipynb index 0ba0d4e5fd..ede5f7b1e1 100644 --- a/examples/hello-world/step-by-step/cifar10/stats/image_stats.ipynb +++ b/examples/hello-world/step-by-step/cifar10/stats/image_stats.ipynb @@ -664,9 +664,8 @@ "\n", "If you would like to see another example of federated statistics calculations and configurations, please checkout [federated_statistics](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/federated-statistics) and [fed_stats with spleen_ct_segmentation](https://github.com/NVIDIA/NVFlare/tree/main/integration/monai/examples/spleen_ct_segmentation_sim)\n", "\n", - "Let's move on to the next example and see how can we train the image classifier using pytorch with CIFAR10 data.\n", - "\n", - "\n" + "Let's move on to the next examples and see how can we train the image classifier using pytorch with CIFAR10 data.\n", + "First we will look at the [sag](../sag/sag.ipynb) example, which illustrates how to use the ScatterAndGather workflow for FedAvg with the Client API.\n" ] } ], diff --git a/examples/hello-world/step-by-step/cifar10/swarm/swarm.ipynb b/examples/hello-world/step-by-step/cifar10/swarm/swarm.ipynb index af5ba361b3..bc66e200f2 100644 --- a/examples/hello-world/step-by-step/cifar10/swarm/swarm.ipynb +++ b/examples/hello-world/step-by-step/cifar10/swarm/swarm.ipynb @@ -154,7 +154,10 @@ "id": "48271064", "metadata": {}, "source": [ - "As an additional resource, also see the [Swarm Learning Example](../../../../advanced/swarm_learning/README.md) which utilizes the CIFAR10 ModelLearner instead of the Client API." + "As an additional resource, also see the [Swarm Learning Example](../../../../advanced/swarm_learning/README.md) which utilizes the CIFAR10 ModelLearner instead of the Client API.\n", + "\n", + "Congratulations! You have completed the CIFAR10 step-by-step example series.\n", + "Next take a look at the [higgs](../../higgs/README.md) example series for how to use machine learning methods for federated learning on tabular data." ] }, { diff --git a/examples/hello-world/step-by-step/higgs/README.md b/examples/hello-world/step-by-step/higgs/README.md index 877a0fb118..9f344b43d3 100644 --- a/examples/hello-world/step-by-step/higgs/README.md +++ b/examples/hello-world/step-by-step/higgs/README.md @@ -1,7 +1,8 @@ # Training traditional ML classifiers with HIGGS dataset -The [HIGGS dataset](https://archive.ics.uci.edu/dataset/280/higgs) contains 11 million instances, each with 28 attributes, for binary classification to predict whether an event corresponds to the decayment of a Higgs boson or not. (Please note that the [UCI's website](https://archive.ics.uci.edu/dataset/280/higgs) may experience occasional downtime) +The [HIGGS dataset](https://archive.ics.uci.edu/dataset/280/higgs) contains 11 million instances, each with 28 attributes, for binary classification to predict whether an event corresponds to the decayment of a Higgs boson or not. Follow the [prepare_data.ipynb](prepare_data.ipynb) notebook to download the HIGGS dataset and prepare the data splits. +(Please note that the [UCI's website](https://archive.ics.uci.edu/dataset/280/higgs) may experience occasional downtime) The first 21 features (columns 2-22) are kinematic properties measured by the particle detectors in the accelerator. The data has been produced using Monte Carlo simulations. The first 21 features are kinematic properties measured by the particle detectors in the accelerator. The last 7 features are functions of the first 21 features; these are high-level features derived by physicists to help discriminate between the two classes. diff --git a/examples/hello-world/step-by-step/higgs/sklearn-kmeans/sklearn_kmeans.ipynb b/examples/hello-world/step-by-step/higgs/sklearn-kmeans/sklearn_kmeans.ipynb index 961ab69c66..1989af4eaa 100644 --- a/examples/hello-world/step-by-step/higgs/sklearn-kmeans/sklearn_kmeans.ipynb +++ b/examples/hello-world/step-by-step/higgs/sklearn-kmeans/sklearn_kmeans.ipynb @@ -452,7 +452,10 @@ "HIGGS dataset is challenging for unsupervised clustering, as we can observe from the result. As shown by the local training with same number of iterations, the score is `model homogeneity_score: 0.0049`. As compared with the FL score of `0.0068`, FL in this case still provides some benefit from the collaborative learning.\n", "\n", "## We are done !\n", - "Congratulations! you have just completed the federated k-Means clustering for tabular data. " + "Congratulations! you have just completed the federated k-Means clustering for tabular data. \n", + "\n", + "Now we will move on from scikit-learn and take a look at how to use federated XGBoost.\n", + "In the next example [xgboost](../xgboost/xgboost_horizontal.ipynb), we will show a federated horizontal xgboost learning with bagging collaboration." ] }, { diff --git a/examples/hello-world/step-by-step/higgs/sklearn-linear/sklearn_linear.ipynb b/examples/hello-world/step-by-step/higgs/sklearn-linear/sklearn_linear.ipynb index 6e653a86bc..462a7448bc 100644 --- a/examples/hello-world/step-by-step/higgs/sklearn-linear/sklearn_linear.ipynb +++ b/examples/hello-world/step-by-step/higgs/sklearn-linear/sklearn_linear.ipynb @@ -454,12 +454,14 @@ "id": "ea7bbacc-b059-4f82-9785-2b22bf840ef9", "metadata": {}, "source": [ - "In this experiment, all three clients have relatively large amount data wiht homogeneous distribution, we would expect the three numbers align within reasonable variation range. \n", + "In this experiment, all three clients have relatively large amount data with homogeneous distribution, we would expect the three numbers align within reasonable variation range. \n", "\n", "The final result for iterative learning is `ending model AUC: 0.6352`, and one-shot learning is `local model AUC: 0.6355`, as compared with FL's `local model AUC: 0.6351`, the numbers do align.\n", "\n", "## We are done !\n", - "Congratulations! you have just completed the federated linear model for tabular data. " + "Congratulations! you have just completed the federated linear model for tabular data. \n", + "\n", + "In the next example [sklearn-svm](../sklearn-svm/sklearn_svm.ipynb), we will demonstrate training a federated SVM model." ] }, { diff --git a/examples/hello-world/step-by-step/higgs/sklearn-svm/sklearn_svm.ipynb b/examples/hello-world/step-by-step/higgs/sklearn-svm/sklearn_svm.ipynb index 29a85b8c44..850cc955c4 100644 --- a/examples/hello-world/step-by-step/higgs/sklearn-svm/sklearn_svm.ipynb +++ b/examples/hello-world/step-by-step/higgs/sklearn-svm/sklearn_svm.ipynb @@ -431,7 +431,9 @@ "The final result for local SVM learning is `model AUC: 0.6217`, as compared with FL's `model AUC: 0.6403`, this confirms our expectation.\n", "\n", "## We are done !\n", - "Congratulations! you have just completed the federated SVM model for tabular data. " + "Congratulations! you have just completed the federated SVM model for tabular data. \n", + "\n", + "In the next example [sklearn-kmeans](../sklearn-kmeans/sklearn_kmeans.ipynb), we will illustrate a federated k-Means clustering." ] }, { diff --git a/examples/hello-world/step-by-step/higgs/stats/tabular_stats.ipynb b/examples/hello-world/step-by-step/higgs/stats/tabular_stats.ipynb index 8946a902ae..455309941c 100644 --- a/examples/hello-world/step-by-step/higgs/stats/tabular_stats.ipynb +++ b/examples/hello-world/step-by-step/higgs/stats/tabular_stats.ipynb @@ -293,7 +293,7 @@ "source": [ "## Create Federated Statistics Job\n", "\n", - "We are going to use NVFLARE job cli to create job. For detailed instructions on Job CLI, please follow the [job cli tutorial](https://github.com/NVIDIA/NVFlare/blob/main/examples/tutorials/job_cli.ipynb)\n", + "We are going to use NVFLARE job cli to create a job. For detailed instructions on Job CLI, please follow the [job cli tutorial](https://github.com/NVIDIA/NVFlare/blob/main/examples/tutorials/job_cli.ipynb)\n", "\n", "Let's check the available job templates, we are going to use one of the existing job templates and modify it to fit our needs. The job template is nothing but server and client-side job configurations." ] @@ -607,7 +607,10 @@ "## We are done !\n", "Congratulations! you have just completed the federated stats calulation for tabular data. \n", "\n", - "If you would like to see a detailed discussion regarding privacy filtering, please checkout the example in [federated statistics](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/federated-statistics) examples." + "If you would like to see a detailed discussion regarding privacy filtering, please checkout the example in [federated statistics](https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/federated-statistics) examples.\n", + "\n", + "Let's move on to the next examples and see how can we use scikit-learn to train federated models on tabular data.\n", + "First we will look at the [sklearn-linear](../sklearn-linear/sklearn_linear.ipynb) example, which illustrates how to train a federated linear model (logistic regression on binary classification)." ] }, { diff --git a/examples/hello-world/step-by-step/higgs/xgboost/xgboost_horizontal.ipynb b/examples/hello-world/step-by-step/higgs/xgboost/xgboost_horizontal.ipynb index 6b1702d5ca..54b35705ab 100644 --- a/examples/hello-world/step-by-step/higgs/xgboost/xgboost_horizontal.ipynb +++ b/examples/hello-world/step-by-step/higgs/xgboost/xgboost_horizontal.ipynb @@ -481,7 +481,11 @@ "Both oneshot and iterative training schemes yield idential results of `local model AUC: 0.81928`. As compared with FL's `global model AUC: 0.82085`, we can notice FL gives some benefits, even under homogeneous data distribution across clients.\n", "\n", "## We are done !\n", - "Congratulations! you have just completed the federated xgboost model for tabular data. " + "Congratulations! you have just completed the federated xgboost model for tabular data. \n", + "\n", + "You have now completed the HIGGS step-by-step example series.\n", + "Next either take a look at the [cifar10](../../cifar10/README.md) example series for how to train an image classifier with PyTorch, or explore the\n", + "[examples/advanced](../../../../advanced/README.md) directory for more in-depth examples." ] }, { diff --git a/examples/tutorials/job_cli.ipynb b/examples/tutorials/job_cli.ipynb index d57d855f90..858c199069 100644 --- a/examples/tutorials/job_cli.ipynb +++ b/examples/tutorials/job_cli.ipynb @@ -15,7 +15,9 @@ "tags": [] }, "source": [ - "In this notebook, we will go through the different commands of the Job CLI to show the syntax and usage of each.\n" + "In this notebook, we will go through the different commands of the Job CLI to show the syntax and usage of each.\n", + "Refer to the [Job CLI Documentation](https://nvflare.readthedocs.io/en/main/user_guide/nvflare_cli/job_cli.html) for more details.\n", + "\n" ] }, { diff --git a/integration/monai/README.md b/integration/monai/README.md index e8ef9a2be2..f1803fba72 100644 --- a/integration/monai/README.md +++ b/integration/monai/README.md @@ -9,10 +9,6 @@ Add `ClientAlgoExecutor` class to allow using MONAI's `ClientAlgo` class in fede Allow the use of bundles from the MONAI [model zoo](https://github.com/Project-MONAI/model-zoo) or custom configurations with NVFlare. -### Non-goals: - -n/a - ## Background MONAI allows the definition of AI models using the "[bundle](https://docs.monai.io/en/latest/bundle.html)" concept. It allows for easy experimentation and sharing of models that have been developed using MONAI. diff --git a/integration/nemo/examples/peft/README.md b/integration/nemo/examples/peft/README.md index bab36e0487..762173e230 100644 --- a/integration/nemo/examples/peft/README.md +++ b/integration/nemo/examples/peft/README.md @@ -25,6 +25,7 @@ docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 -- For easy experimentation with NeMo, install NVFlare and mount the code inside the [nemo_nvflare](./nemo_nvflare) folder. ``` +cd nemo_nvflare pip install nvflare~=2.4.0rc7 export PYTHONPATH=${PYTHONPATH}:/workspace ``` @@ -34,6 +35,7 @@ export PYTHONPATH=${PYTHONPATH}:/workspace We use [JupyterLab](https://jupyterlab.readthedocs.io) for this example. To start JupyterLab, run ``` +cd /workspace jupyter lab . ``` and open [peft.ipynb](./peft.ipynb). diff --git a/integration/nemo/examples/prompt_learning/README.md b/integration/nemo/examples/prompt_learning/README.md index c25457496e..b7e1049045 100644 --- a/integration/nemo/examples/prompt_learning/README.md +++ b/integration/nemo/examples/prompt_learning/README.md @@ -25,6 +25,7 @@ docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 -- For easy experimentation with NeMo, install NVFlare and mount the code inside the [nemo_nvflare](./nemo_nvflare) folder. ``` +cd nemo_nvflare pip install nvflare~=2.4.0rc7 export PYTHONPATH=${PYTHONPATH}:/workspace ``` @@ -34,6 +35,7 @@ export PYTHONPATH=${PYTHONPATH}:/workspace We use [JupyterLab](https://jupyterlab.readthedocs.io) for this example. To start JupyterLab, run ``` +cd /workspace jupyter lab . ``` and open [prompt_learning.ipynb](./prompt_learning.ipynb). diff --git a/integration/nemo/examples/prompt_learning/prompt_learning.ipynb b/integration/nemo/examples/prompt_learning/prompt_learning.ipynb index 4d76fcd860..66f80a2dc9 100644 --- a/integration/nemo/examples/prompt_learning/prompt_learning.ipynb +++ b/integration/nemo/examples/prompt_learning/prompt_learning.ipynb @@ -14,7 +14,13 @@ "The prompt learning technique shown in the example is [p-tuning](https://arxiv.org/abs/2103.10385), which adds a small prompt encoder network to the LLM\n", "to produce virtual token embeddings that guide the model toward the desired output of the downstream task.\n", "\n", - "For more details on how to change hyperparameters for prompt learning in NeMo, see this [tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Multitask_Prompt_and_PTuning.ipynb) which is also the basis for this NVFlare tutorial." + "For more details on how to change hyperparameters for prompt learning in NeMo, see this [tutorial](https://github.com/NVIDIA/NeMo/blob/main/tutorials/nlp/Multitask_Prompt_and_PTuning.ipynb) which is also the basis for this NVFlare tutorial.\n", + "\n", + "\n", + "\n", + "In our federated implementation, the LLM parameters stay fixed. Prompt encoder parameters are trained/updated and averaged on the server.\n", + "\n", + "" ] }, { diff --git a/integration/nemo/examples/supervised_fine_tuning/README.md b/integration/nemo/examples/supervised_fine_tuning/README.md index 610814d3d8..f34a4a8539 100644 --- a/integration/nemo/examples/supervised_fine_tuning/README.md +++ b/integration/nemo/examples/supervised_fine_tuning/README.md @@ -24,6 +24,7 @@ docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 -- For easy experimentation with NeMo, install NVFlare and mount the code inside the [nemo_nvflare](./nemo_nvflare) folder. ``` +cd nemo_nvflare pip install nvflare~=2.4.0rc7 export PYTHONPATH=${PYTHONPATH}:/workspace ``` diff --git a/job_templates/readme.md b/job_templates/readme.md index 6963c6165f..245ce3c198 100644 --- a/job_templates/readme.md +++ b/job_templates/readme.md @@ -13,11 +13,13 @@ Each job template contains the following informations * information card: info.md for display purpose * information config: used by program +Refer to the [Job CLI Documentation](https://nvflare.readthedocs.io/en/main/user_guide/nvflare_cli/job_cli.html) for details on how to use the Job Templates with the Job CLI. + ## Configuration format Configurations are written in HOCON (human optimized object Notation). As a variant of JSON, .conf can also use json format. The pyhocon format allows for comments, and you can remove many of the double quotes as well as replace ":" with "=" to make the configurations look cleaner. -You can find details in [pyhoconb: HOCON Parser for python](https://github.com/chimpler/pyhocon). +You can find details in [pyhocon: HOCON Parser for python](https://github.com/chimpler/pyhocon). ## List of Job Templates From 899f11e8f6b7b071c4d5487c5e83157515521aea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Fri, 2 Feb 2024 11:58:53 -0800 Subject: [PATCH 30/39] Add notes about running TF with GPUs (#2348) --- examples/hello-world/hello-cyclic/README.md | 28 +++++++++++++++++-- examples/hello-world/hello-tf2/README.md | 30 ++++++++++++++++++--- examples/hello-world/ml-to-fl/tf/README.md | 27 ++++++++++++++++--- 3 files changed, 76 insertions(+), 9 deletions(-) diff --git a/examples/hello-world/hello-cyclic/README.md b/examples/hello-world/hello-cyclic/README.md index 84d81eda1f..60c8c632ea 100644 --- a/examples/hello-world/hello-cyclic/README.md +++ b/examples/hello-world/hello-cyclic/README.md @@ -27,8 +27,8 @@ bash ./prepare_data.sh Use nvflare simulator to run the hello-examples: -``` -nvflare simulator -w /tmp/nvflare/ -n 2 -t 2 hello-cyclic/jobs/hello-cyclic +```bash +nvflare simulator -w /tmp/nvflare/ -n 2 -t 2 ./jobs/hello-cyclic ``` ### 3. Access the logs and results @@ -40,3 +40,27 @@ $ ls /tmp/nvflare/simulate_job/ app_server app_site-1 app_site-2 log.txt ``` + +### 4. Notes on running with GPUs + +For running with GPUs, we recommend using +[NVIDIA TensorFlow docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) + +If you choose to run the example using GPUs, it is important to note that, +by default, TensorFlow will attempt to allocate all available GPU memory at the start. +In scenarios where multiple clients are involved, you have a couple of options to address this. + +One approach is to include specific flags to prevent TensorFlow from allocating all GPU memory. +For instance: + +```bash +TF_FORCE_GPU_ALLOW_GROWTH=true nvflare simulator -w /tmp/nvflare/ -n 2 -t 2 ./jobs/hello-cyclic +``` + +If you possess more GPUs than clients, +an alternative strategy is to run one client on each GPU. +This can be achieved as illustrated below: + +```bash +TF_FORCE_GPU_ALLOW_GROWTH=true nvflare simulator -w /tmp/nvflare/ -n 2 -gpu 0,1 ./jobs/hello-cyclic +``` diff --git a/examples/hello-world/hello-tf2/README.md b/examples/hello-world/hello-tf2/README.md index 5bdeb6a0e5..3bad827f25 100644 --- a/examples/hello-world/hello-tf2/README.md +++ b/examples/hello-world/hello-tf2/README.md @@ -26,10 +26,10 @@ Prepare the data first: bash ./prepare_data.sh ``` -Use nvflare simulator to run the hello-examples: (TF2 does not allow multiple processes to be running on a single GPU at the same time. Need to set the simulator threads to 1. "-gpu" option can be used to run multiple concurrent clients.) +Use nvflare simulator to run the hello-examples: -``` -nvflare simulator -w /tmp/nvflare/ -n 2 -t 1 hello-tf2/jobs/hello-tf2 +```bash +nvflare simulator -w /tmp/nvflare/ -n 2 -t 2 ./jobs/hello-tf2 ``` ### 3. Access the logs and results @@ -41,3 +41,27 @@ $ ls /tmp/nvflare/simulate_job/ app_server app_site-1 app_site-2 log.txt ``` + +### 4. Notes on running with GPUs + +For running with GPUs, we recommend using +[NVIDIA TensorFlow docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) + +If you choose to run the example using GPUs, it is important to note that, +by default, TensorFlow will attempt to allocate all available GPU memory at the start. +In scenarios where multiple clients are involved, you have a couple of options to address this. + +One approach is to include specific flags to prevent TensorFlow from allocating all GPU memory. +For instance: + +```bash +TF_FORCE_GPU_ALLOW_GROWTH=true nvflare simulator -w /tmp/nvflare/ -n 2 -t 2 ./jobs/hello-tf2 +``` + +If you possess more GPUs than clients, +an alternative strategy is to run one client on each GPU. +This can be achieved as illustrated below: + +```bash +TF_FORCE_GPU_ALLOW_GROWTH=true nvflare simulator -w /tmp/nvflare/ -n 2 -gpu 0,1 ./jobs/hello-tf2 +``` diff --git a/examples/hello-world/ml-to-fl/tf/README.md b/examples/hello-world/ml-to-fl/tf/README.md index e2444844f4..ce3845dfe9 100644 --- a/examples/hello-world/ml-to-fl/tf/README.md +++ b/examples/hello-world/ml-to-fl/tf/README.md @@ -31,8 +31,7 @@ nvflare job list_templates \* depends on whether TF can found GPU or not -Note that for running with GPUs, we recommend using [NVIDIA TensorFlow docker](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/tensorflow) - +For running with GPUs, please check the [note](#notes-on-running-with-gpus) ## Transform CIFAR10 TensorFlow training code to FL with NVFLARE Client API @@ -108,7 +107,27 @@ Then we can run the job using the simulator: ```bash bash ./prepare_data.sh -TF_GPU_ALLOCATOR=cuda_malloc_async nvflare simulator -n 2 -t 2 ./jobs/tensorflow_multi_gpu -w tensorflow_multi_gpu_workspace +nvflare simulator -n 2 -t 2 ./jobs/tensorflow_multi_gpu -w tensorflow_multi_gpu_workspace ``` -Note that the flag "TF_GPU_ALLOCATOR=cuda_malloc_async" is only needed if you are going to run more than one process in the same GPU. +## Notes on running with GPUs + + +If you choose to run the example using GPUs, it is important to note that, +by default, TensorFlow will attempt to allocate all available GPU memory at the start. +In scenarios where multiple clients are involved, you have a couple of options to address this. + +One approach is to include specific flags to prevent TensorFlow from allocating all GPU memory. +For instance: + +```bash +TF_FORCE_GPU_ALLOW_GROWTH=true TF_GPU_ALLOCATOR=cuda_malloc_async nvflare simulator -n 2 -t 2 ./jobs/tensorflow_multi_gpu -w tensorflow_multi_gpu_workspace +``` + +If you possess more GPUs than clients, +an alternative strategy is to run one client on each GPU. +This can be achieved as illustrated below: + +```bash +nvflare simulator -n 2 -gpu 0,1 ./jobs/tensorflow_multi_gpu -w tensorflow_multi_gpu_workspace +``` From 2f9d00dda2f8d3b4044694a2585833c730fbf203 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Fri, 2 Feb 2024 13:53:28 -0800 Subject: [PATCH 31/39] Fix tb receiver (#2349) Co-authored-by: Chester Chen <512707+chesterxgchen@users.noreply.github.com> --- nvflare/apis/analytix.py | 6 +-- nvflare/app_opt/tracking/tb/tb_receiver.py | 59 +++++++++++++++++----- 2 files changed, 47 insertions(+), 18 deletions(-) diff --git a/nvflare/apis/analytix.py b/nvflare/apis/analytix.py index 68772b13a6..0cb0ab1b45 100644 --- a/nvflare/apis/analytix.py +++ b/nvflare/apis/analytix.py @@ -183,11 +183,7 @@ def convert_data_type( return sender_data_type if sender == LogWriterName.MLFLOW and receiver == LogWriterName.TORCH_TB: - if AnalyticsDataType.PARAMETER == sender_data_type: - return AnalyticsDataType.SCALAR - elif AnalyticsDataType.PARAMETERS == sender_data_type: - return AnalyticsDataType.SCALARS - elif AnalyticsDataType.METRIC == sender_data_type: + if AnalyticsDataType.METRIC == sender_data_type: return AnalyticsDataType.SCALAR elif AnalyticsDataType.METRICS == sender_data_type: return AnalyticsDataType.SCALARS diff --git a/nvflare/app_opt/tracking/tb/tb_receiver.py b/nvflare/app_opt/tracking/tb/tb_receiver.py index d30b2b94c0..585087dadc 100644 --- a/nvflare/app_opt/tracking/tb/tb_receiver.py +++ b/nvflare/app_opt/tracking/tb/tb_receiver.py @@ -28,13 +28,22 @@ AnalyticsDataType.TEXT: "add_text", AnalyticsDataType.IMAGE: "add_image", AnalyticsDataType.SCALARS: "add_scalars", - AnalyticsDataType.PARAMETER: "add_scalar", - AnalyticsDataType.PARAMETERS: "add_scalars", AnalyticsDataType.METRIC: "add_scalar", AnalyticsDataType.METRICS: "add_scalars", } +def _create_new_data(key, value, sender): + if isinstance(value, (int, float)): + data_type = AnalyticsDataType.SCALAR + elif isinstance(value, str): + data_type = AnalyticsDataType.TEXT + else: + return None + + return AnalyticsData(key=key, value=value, data_type=data_type, sender=sender) + + class TBAnalyticsReceiver(AnalyticsReceiver): def __init__(self, tb_folder="tb_events", events: Optional[List[str]] = None): """Receives analytics data to save to TensorBoard. @@ -71,6 +80,27 @@ def initialize(self, fl_ctx: FLContext): os.makedirs(root_log_dir, exist_ok=True) self.root_log_dir = root_log_dir + def _convert_to_records(self, analytic_data: AnalyticsData, fl_ctx: FLContext) -> List[AnalyticsData]: + # break dict of stuff to smaller items to support + # AnalyticsDataType.PARAMETER and AnalyticsDataType.PARAMETERS + records = [] + + if analytic_data.data_type in (AnalyticsDataType.PARAMETER, AnalyticsDataType.PARAMETERS): + for k, v in ( + analytic_data.value.items() + if analytic_data.data_type == AnalyticsDataType.PARAMETERS + else [(analytic_data.tag, analytic_data.value)] + ): + new_data = _create_new_data(k, v, analytic_data.sender) + if new_data is None: + self.log_warning(fl_ctx, f"Entry {k} of type {type(v)} is not supported.", fire_event=False) + else: + records.append(new_data) + else: + records.append(analytic_data) + + return records + def save(self, fl_ctx: FLContext, shareable: Shareable, record_origin): dxo = from_shareable(shareable) analytic_data = AnalyticsData.from_dxo(dxo) @@ -86,19 +116,22 @@ def save(self, fl_ctx: FLContext, shareable: Shareable, record_origin): # do different things depending on the type in dxo self.log_debug( fl_ctx, - f"save data {analytic_data} from {record_origin}", + f"try to save data {analytic_data} from {record_origin}", fire_event=False, ) - func_name = FUNCTION_MAPPING.get(analytic_data.data_type, None) - if func_name is None: - self.log_error(fl_ctx, f"The data_type {analytic_data.data_type} is not supported.", fire_event=False) - return - - func = getattr(writer, func_name) - if analytic_data.step: - func(analytic_data.tag, analytic_data.value, analytic_data.step) - else: - func(analytic_data.tag, analytic_data.value) + data_records = self._convert_to_records(analytic_data, fl_ctx) + + for data_record in data_records: + func_name = FUNCTION_MAPPING.get(data_record.data_type, None) + if func_name is None: + self.log_warning(fl_ctx, f"The data_type {data_record.data_type} is not supported.", fire_event=False) + return + + func = getattr(writer, func_name) + if data_record.step: + func(data_record.tag, data_record.value, data_record.step) + else: + func(data_record.tag, data_record.value) def finalize(self, fl_ctx: FLContext): for writer in self.writers_table.values(): From dbdbdeba22e3657f6c46e148b562c82a137c6b75 Mon Sep 17 00:00:00 2001 From: Sean Yang Date: Mon, 5 Feb 2024 13:53:18 -0800 Subject: [PATCH 32/39] clarify nemo example readmes (#2352) --- integration/nemo/examples/peft/README.md | 5 ++--- integration/nemo/examples/prompt_learning/README.md | 5 ++--- integration/nemo/examples/supervised_fine_tuning/README.md | 4 ++-- 3 files changed, 6 insertions(+), 8 deletions(-) diff --git a/integration/nemo/examples/peft/README.md b/integration/nemo/examples/peft/README.md index 762173e230..01d116272d 100644 --- a/integration/nemo/examples/peft/README.md +++ b/integration/nemo/examples/peft/README.md @@ -16,8 +16,9 @@ In the following, we assume this example folder of the container is mounted to ` > Note in the following, mount both the [current directory](./) and the [job_templates](../../../../job_templates) > directory to locations inside the docker container. Please make sure you have cloned the full NVFlare repo. -Start the docker container using +Start the docker container from **this directory** using ``` +# cd NVFlare/integration/nemo/examples/peft DOCKER_IMAGE="nvcr.io/nvidia/nemo:23.10" docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 \ -v ${PWD}/../../../../job_templates:/job_templates -v ${PWD}:/workspace -w /workspace ${DOCKER_IMAGE} @@ -25,7 +26,6 @@ docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 -- For easy experimentation with NeMo, install NVFlare and mount the code inside the [nemo_nvflare](./nemo_nvflare) folder. ``` -cd nemo_nvflare pip install nvflare~=2.4.0rc7 export PYTHONPATH=${PYTHONPATH}:/workspace ``` @@ -35,7 +35,6 @@ export PYTHONPATH=${PYTHONPATH}:/workspace We use [JupyterLab](https://jupyterlab.readthedocs.io) for this example. To start JupyterLab, run ``` -cd /workspace jupyter lab . ``` and open [peft.ipynb](./peft.ipynb). diff --git a/integration/nemo/examples/prompt_learning/README.md b/integration/nemo/examples/prompt_learning/README.md index b7e1049045..0e2125a92f 100644 --- a/integration/nemo/examples/prompt_learning/README.md +++ b/integration/nemo/examples/prompt_learning/README.md @@ -16,8 +16,9 @@ In our federated implementation, the LLM parameters stay fixed. Prompt encoder p The example was tested with the [NeMo 23.02 container](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo). In the following, we assume this example folder of the container is mounted to `/workspace` and all downloading, etc. operations are based on this root path. -Start the docker container using +Start the docker container from **this directory** using ``` +# cd NVFlare/integration/nemo/examples/prompt_learning DOCKER_IMAGE="nvcr.io/nvidia/nemo:23.02" docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 \ -v ${PWD}:/workspace -w /workspace ${DOCKER_IMAGE} @@ -25,7 +26,6 @@ docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 -- For easy experimentation with NeMo, install NVFlare and mount the code inside the [nemo_nvflare](./nemo_nvflare) folder. ``` -cd nemo_nvflare pip install nvflare~=2.4.0rc7 export PYTHONPATH=${PYTHONPATH}:/workspace ``` @@ -35,7 +35,6 @@ export PYTHONPATH=${PYTHONPATH}:/workspace We use [JupyterLab](https://jupyterlab.readthedocs.io) for this example. To start JupyterLab, run ``` -cd /workspace jupyter lab . ``` and open [prompt_learning.ipynb](./prompt_learning.ipynb). diff --git a/integration/nemo/examples/supervised_fine_tuning/README.md b/integration/nemo/examples/supervised_fine_tuning/README.md index f34a4a8539..4fb829ea7d 100644 --- a/integration/nemo/examples/supervised_fine_tuning/README.md +++ b/integration/nemo/examples/supervised_fine_tuning/README.md @@ -15,8 +15,9 @@ The example was tested using the [NeMo Docker container](https://catalog.ngc.nvi available with `docker pull nvcr.io/nvidia/nemo:23.06`. In the following, we assume this example folder of the container is mounted to `/workspace` and all downloading, etc. operations are based on this root path. -Start the docker container using +Start the docker container from **this directory** using ``` +# cd NVFlare/integration/nemo/examples/supervised_fine_tuning DOCKER_IMAGE="nvcr.io/nvidia/nemo:23.06" docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 --ulimit memlock=-1 --ulimit stack=67108864 \ -v ${PWD}:/workspace -w /workspace ${DOCKER_IMAGE} @@ -24,7 +25,6 @@ docker run --runtime=nvidia -it --rm --shm-size=16g -p 8888:8888 -p 6006:6006 -- For easy experimentation with NeMo, install NVFlare and mount the code inside the [nemo_nvflare](./nemo_nvflare) folder. ``` -cd nemo_nvflare pip install nvflare~=2.4.0rc7 export PYTHONPATH=${PYTHONPATH}:/workspace ``` From 8cb03edd51d87b7c13cf87e7c4b4925ce8e83091 Mon Sep 17 00:00:00 2001 From: Chester Chen <512707+chesterxgchen@users.noreply.github.com> Date: Mon, 5 Feb 2024 14:37:20 -0800 Subject: [PATCH 33/39] DataBus (#2285) * add message bus address PR comments formatting * Check invalid input directory in nvflare config (#2295) * check invalid input directory * check invalid input directory add doc string add doc string rename receive_messages() to receive_message() change doc str to google doc string style rebase and formats * remove space * restore space * Address PR comments 1) remove data store scope/topic 2) add pub_sub interface and let databus implements the inferface 3) remove function_utils.py and unit tests for another PR * Address PR comments 1) remove data store scope/topic 2) add pub_sub interface and let databus implements the inferface 3) remove function_utils.py and unit tests for another PR * reduce lock scope * make sure the publish in parallel instead of sequential * rename send_data/receive_data() to put_data()/get_data() --------- Co-authored-by: Sean Yang --- nvflare/fuel/data_event/__init__.py | 13 +++ nvflare/fuel/data_event/data_bus.py | 106 ++++++++++++++++++ nvflare/fuel/data_event/event_manager.py | 45 ++++++++ nvflare/fuel/data_event/pub_sub.py | 34 ++++++ runtest.sh | 2 +- tests/unit_test/fuel/data_event/__init__.py | 13 +++ .../fuel/data_event/data_bus_test.py | 86 ++++++++++++++ 7 files changed, 298 insertions(+), 1 deletion(-) create mode 100644 nvflare/fuel/data_event/__init__.py create mode 100644 nvflare/fuel/data_event/data_bus.py create mode 100644 nvflare/fuel/data_event/event_manager.py create mode 100644 nvflare/fuel/data_event/pub_sub.py create mode 100644 tests/unit_test/fuel/data_event/__init__.py create mode 100644 tests/unit_test/fuel/data_event/data_bus_test.py diff --git a/nvflare/fuel/data_event/__init__.py b/nvflare/fuel/data_event/__init__.py new file mode 100644 index 0000000000..d9155f923f --- /dev/null +++ b/nvflare/fuel/data_event/__init__.py @@ -0,0 +1,13 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/nvflare/fuel/data_event/data_bus.py b/nvflare/fuel/data_event/data_bus.py new file mode 100644 index 0000000000..ef1fd4a4be --- /dev/null +++ b/nvflare/fuel/data_event/data_bus.py @@ -0,0 +1,106 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import threading +from concurrent.futures import ThreadPoolExecutor +from typing import Any, Callable, List + +from nvflare.fuel.data_event.pub_sub import EventPubSub + + +class DataBus(EventPubSub): + """ + Singleton class for a simple data bus implementation. + + This class allows components to subscribe to topics, publish messages to topics, + and store/retrieve messages associated with specific keys and topics. + """ + + _instance = None + _lock = threading.Lock() + + def __new__(cls) -> "DataBus": + """ + Create a new instance of the DataBus class. + This method ensures that only one instance of the class is created (singleton pattern). + The databus + + + """ + with cls._lock: + if not cls._instance: + cls._instance = super(DataBus, cls).__new__(cls) + cls._instance.subscribers = {} + cls._instance.data_store = {} + return cls._instance + + def subscribe(self, topics: List[str], callback: Callable[[str, Any, "DataBus"], None]) -> None: + """ + Subscribe a callback function to one or more topics. + + Args: + topics (List[str]): A list of topics to subscribe to. + callback (Callable): The callback function to be called when messages are published to the subscribed topics. + """ + + if not topics: + raise ValueError("topics must non-empty") + + for topic in topics: + if topic.isspace(): + raise ValueError(f"topics {topics}contains white space topic") + + with self._lock: + if topic not in self.subscribers: + self.subscribers[topic] = [] + self.subscribers[topic].append(callback) + + def publish(self, topics: List[str], datum: Any) -> None: + """ + Publish a data to one or more topics, notifying all subscribed callbacks. + + Args: + topics (List[str]): A list of topics to publish the data to. + datum (Any): The data to be published to the specified topics. + """ + if topics: + for topic in topics: + if topic in self.subscribers: + with self._lock: + executor = ThreadPoolExecutor(max_workers=len(self.subscribers[topic])) + for callback in self.subscribers[topic]: + executor.submit(callback, topic, datum, self) + executor.shutdown() + + def put_data(self, key: Any, datum: Any) -> None: + """ + Store a data associated with a key and topic. + + Args: + key (Any): The key to associate with the stored message. + datum (Any): The message to be stored. + """ + with self._lock: + self.data_store[key] = datum + + def get_data(self, key: Any) -> Any: + """ + Retrieve a stored data associated with a key and topic. + + Args: + key (Any): The key associated with the stored message. + + Returns: + Any: The stored datum if found, or None if not found. + """ + return self.data_store.get(key) diff --git a/nvflare/fuel/data_event/event_manager.py b/nvflare/fuel/data_event/event_manager.py new file mode 100644 index 0000000000..6421f8bf4e --- /dev/null +++ b/nvflare/fuel/data_event/event_manager.py @@ -0,0 +1,45 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Any, Optional + +from nvflare.fuel.data_event.data_bus import DataBus + + +class EventManager: + """ + Class for managing events by interacting with a DataBus. + + Args: + data_bus (DataBus): An instance of the DataBus class used for event communication. + """ + + def __init__(self, data_bus: "DataBus"): + """ + Initialize the EventManager with a DataBus instance. + + Args: + data_bus (DataBus): An instance of the DataBus class used for event communication. + """ + self.data_bus = data_bus + + def fire_event(self, event_name: str, event_data: Optional[Any] = None) -> None: + """ + Fire an event by publishing it to the DataBus. + + Args: + event_name (str): The name of the event to be fired. + event_data (Any, optional): Additional data associated with the event (default is None). + """ + self.data_bus.publish([event_name], event_data) diff --git a/nvflare/fuel/data_event/pub_sub.py b/nvflare/fuel/data_event/pub_sub.py new file mode 100644 index 0000000000..63583c8b13 --- /dev/null +++ b/nvflare/fuel/data_event/pub_sub.py @@ -0,0 +1,34 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from typing import Any, Callable, List + + +class EventPubSub: + def subscribe(self, topics: List[str], callback: Callable[[str, Any, "DataBus"], None]) -> None: + """ + Subscribe a callback function to one or more topics. + + Args: + topics (List[str]): A list of topics to subscribe to. + callback (Callable): The callback function to be called when messages are published to the subscribed topics. + """ + + def publish(self, topics: List[str], datum: Any) -> None: + """ + Publish a message to one or more topics, notifying all subscribed callbacks. + + Args: + topics (List[str]): A list of topics to publish the message to. + datum (Any): The message to be published to the specified topics. + """ diff --git a/runtest.sh b/runtest.sh index e7644cf736..b7df3b6be1 100755 --- a/runtest.sh +++ b/runtest.sh @@ -92,7 +92,7 @@ function check_license() { folders_to_check_license="nvflare examples tests integration research" echo "checking license header in folder: $folders_to_check_license" (grep -r --include "*.py" --exclude-dir "*protos*" -L \ - "\(# Copyright (c) \(2021\|2022\|2023\), NVIDIA CORPORATION. All rights reserved.\)\|\(This file is released into the public domain.\)" \ + "\(# Copyright (c) \(2021\|2022\|2023\|2024\), NVIDIA CORPORATION. All rights reserved.\)\|\(This file is released into the public domain.\)" \ ${folders_to_check_license} || true) > no_license.lst if [ -s no_license.lst ]; then # The file is not-empty. diff --git a/tests/unit_test/fuel/data_event/__init__.py b/tests/unit_test/fuel/data_event/__init__.py new file mode 100644 index 0000000000..d9155f923f --- /dev/null +++ b/tests/unit_test/fuel/data_event/__init__.py @@ -0,0 +1,13 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/tests/unit_test/fuel/data_event/data_bus_test.py b/tests/unit_test/fuel/data_event/data_bus_test.py new file mode 100644 index 0000000000..4979688d13 --- /dev/null +++ b/tests/unit_test/fuel/data_event/data_bus_test.py @@ -0,0 +1,86 @@ +# Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import unittest + +from nvflare.fuel.data_event.data_bus import DataBus +from nvflare.fuel.data_event.event_manager import EventManager + + +class TestMessageBus(unittest.TestCase): + def setUp(self): + self.data_bus = DataBus() + self.event_manager = EventManager(self.data_bus) + + def test_subscribe_and_publish(self): + result = {"count": 0} + + def callback_function(topic, datum, data_bus): + result["count"] += 1 + + self.data_bus.subscribe(["test_topic"], callback_function) + self.data_bus.publish(["test_topic"], "Test Message 1") + self.data_bus.publish(["test_topic"], "Test Message 2") + + self.assertEqual(result["count"], 2) + + def test_singleton_message_bus(self): + data_bus1 = DataBus() + data_bus1.put_data("user_1", "Hello from User 1!") + user_1_message = data_bus1.get_data("user_1") + self.assertEqual(user_1_message, "Hello from User 1!") + + message_bus2 = DataBus() + user_1_message = message_bus2.get_data("user_1") + self.assertEqual(user_1_message, "Hello from User 1!") + + def test_send_message_and_receive_messages(self): + self.data_bus.put_data("user_1", "Hello from User 1!") + self.data_bus.put_data("user_2", "Greetings from User 2!") + + user_1_message = self.data_bus.get_data("user_1") + user_2_message = self.data_bus.get_data("user_2") + + self.assertEqual(user_1_message, "Hello from User 1!") + self.assertEqual(user_2_message, "Greetings from User 2!") + + self.data_bus.put_data("user_1", "2nd greetings from User 1!") + user_1_message = self.data_bus.get_data("user_1") + self.assertEqual(user_1_message, "2nd greetings from User 1!") + + def test_send_message_and_receive_messages_abnormal(self): + user_3_message = self.data_bus.get_data("user_3") + self.assertEqual(user_3_message, None) + + def test_fire_event(self): + + result = { + "test_event": {"event_received": False}, + "dev_event": {"event_received": False}, + "prod_event": {"event_received": False}, + } + + def event_handler(topic, data, data_bus): + result[topic]["event_received"] = True + if data_bus.get_data("hi") == "hello": + self.data_bus.put_data("hi", "hello-world") + + self.data_bus.put_data("hi", "hello") + + self.data_bus.subscribe(["test_event", "dev_event", "prod_event"], event_handler) + self.event_manager.fire_event("test_event", {"key": "value"}) + self.event_manager.fire_event("dev_event", {"key": "value"}) + + self.assertTrue(result["test_event"]["event_received"]) + self.assertTrue(result["dev_event"]["event_received"]) + self.assertFalse(result["prod_event"]["event_received"]) From 912adb96f7d410558b92a29082f4212a83a5182d Mon Sep 17 00:00:00 2001 From: Yan Cheng <58191769+yanchengnv@users.noreply.github.com> Date: Wed, 7 Feb 2024 10:22:10 -0500 Subject: [PATCH 34/39] Native FLARE/XGB integration (#2354) * fed xgb * support c integration * implement c server bridge * add docstring; change bridge to adaptor * integrate with xgb federated * remove unused import * reorganize * fix format * added in_process support flag * add license text * fix fstring * change task timeout back to int * add docstr; removed unused code --- nvflare/app_common/xgb/__init__.py | 18 + nvflare/app_common/xgb/adaptors/__init__.py | 13 + nvflare/app_common/xgb/adaptors/adaptor.py | 431 +++++++++++++ .../xgb/adaptors/grpc_client_adaptor.py | 253 ++++++++ .../xgb/adaptors/grpc_server_adaptor.py | 158 +++++ nvflare/app_common/xgb/controller.py | 609 ++++++++++++++++++ nvflare/app_common/xgb/data_loader.py | 30 + nvflare/app_common/xgb/defs.py | 95 +++ nvflare/app_common/xgb/executor.py | 171 +++++ nvflare/app_common/xgb/fed_controller.py | 62 ++ nvflare/app_common/xgb/fed_executor.py | 63 ++ nvflare/app_common/xgb/grpc_client.py | 178 +++++ nvflare/app_common/xgb/grpc_server.py | 82 +++ nvflare/app_common/xgb/mock/__init__.py | 13 + nvflare/app_common/xgb/mock/aggr_servicer.py | 122 ++++ .../app_common/xgb/mock/mock_client_runner.py | 123 ++++ .../app_common/xgb/mock/mock_controller.py | 61 ++ nvflare/app_common/xgb/mock/mock_executor.py | 43 ++ .../app_common/xgb/mock/mock_server_runner.py | 46 ++ nvflare/app_common/xgb/mock/run_client.py | 108 ++++ nvflare/app_common/xgb/mock/run_server.py | 43 ++ nvflare/app_common/xgb/proto/__init__.py | 13 + nvflare/app_common/xgb/proto/federated.proto | 85 +++ nvflare/app_common/xgb/proto/federated_pb2.py | 59 ++ .../app_common/xgb/proto/federated_pb2.pyi | 100 +++ .../xgb/proto/federated_pb2_grpc.py | 179 +++++ nvflare/app_common/xgb/proto/gen_proto.sh | 1 + nvflare/app_common/xgb/runners/__init__.py | 13 + .../xgb/runners/xgb_client_runner.py | 141 ++++ nvflare/app_common/xgb/runners/xgb_runner.py | 63 ++ .../xgb/runners/xgb_server_runner.py | 42 ++ nvflare/app_common/xgb/sender.py | 89 +++ nvflare/app_common/xgb/tb.py | 36 ++ nvflare/app_common/xgb/xgb_params.py | 29 + 34 files changed, 3572 insertions(+) create mode 100644 nvflare/app_common/xgb/__init__.py create mode 100644 nvflare/app_common/xgb/adaptors/__init__.py create mode 100644 nvflare/app_common/xgb/adaptors/adaptor.py create mode 100644 nvflare/app_common/xgb/adaptors/grpc_client_adaptor.py create mode 100644 nvflare/app_common/xgb/adaptors/grpc_server_adaptor.py create mode 100644 nvflare/app_common/xgb/controller.py create mode 100644 nvflare/app_common/xgb/data_loader.py create mode 100644 nvflare/app_common/xgb/defs.py create mode 100644 nvflare/app_common/xgb/executor.py create mode 100644 nvflare/app_common/xgb/fed_controller.py create mode 100644 nvflare/app_common/xgb/fed_executor.py create mode 100644 nvflare/app_common/xgb/grpc_client.py create mode 100644 nvflare/app_common/xgb/grpc_server.py create mode 100644 nvflare/app_common/xgb/mock/__init__.py create mode 100644 nvflare/app_common/xgb/mock/aggr_servicer.py create mode 100644 nvflare/app_common/xgb/mock/mock_client_runner.py create mode 100644 nvflare/app_common/xgb/mock/mock_controller.py create mode 100644 nvflare/app_common/xgb/mock/mock_executor.py create mode 100644 nvflare/app_common/xgb/mock/mock_server_runner.py create mode 100644 nvflare/app_common/xgb/mock/run_client.py create mode 100644 nvflare/app_common/xgb/mock/run_server.py create mode 100644 nvflare/app_common/xgb/proto/__init__.py create mode 100644 nvflare/app_common/xgb/proto/federated.proto create mode 100644 nvflare/app_common/xgb/proto/federated_pb2.py create mode 100644 nvflare/app_common/xgb/proto/federated_pb2.pyi create mode 100644 nvflare/app_common/xgb/proto/federated_pb2_grpc.py create mode 100644 nvflare/app_common/xgb/proto/gen_proto.sh create mode 100644 nvflare/app_common/xgb/runners/__init__.py create mode 100644 nvflare/app_common/xgb/runners/xgb_client_runner.py create mode 100644 nvflare/app_common/xgb/runners/xgb_runner.py create mode 100644 nvflare/app_common/xgb/runners/xgb_server_runner.py create mode 100644 nvflare/app_common/xgb/sender.py create mode 100644 nvflare/app_common/xgb/tb.py create mode 100644 nvflare/app_common/xgb/xgb_params.py diff --git a/nvflare/app_common/xgb/__init__.py b/nvflare/app_common/xgb/__init__.py new file mode 100644 index 0000000000..df104c37e9 --- /dev/null +++ b/nvflare/app_common/xgb/__init__.py @@ -0,0 +1,18 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from nvflare.app_common.xgb.fed_controller import XGBFedController +from nvflare.app_common.xgb.fed_executor import FedXGBHistogramExecutor +from nvflare.app_common.xgb.mock.mock_controller import MockXGBController +from nvflare.app_common.xgb.mock.mock_executor import MockXGBExecutor diff --git a/nvflare/app_common/xgb/adaptors/__init__.py b/nvflare/app_common/xgb/adaptors/__init__.py new file mode 100644 index 0000000000..4fc50543f1 --- /dev/null +++ b/nvflare/app_common/xgb/adaptors/__init__.py @@ -0,0 +1,13 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/nvflare/app_common/xgb/adaptors/adaptor.py b/nvflare/app_common/xgb/adaptors/adaptor.py new file mode 100644 index 0000000000..861e196de7 --- /dev/null +++ b/nvflare/app_common/xgb/adaptors/adaptor.py @@ -0,0 +1,431 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import threading +import time +from abc import ABC, abstractmethod + +from nvflare.apis.fl_component import FLComponent +from nvflare.apis.fl_context import FLContext +from nvflare.apis.shareable import Shareable +from nvflare.apis.signal import Signal +from nvflare.app_common.xgb.defs import Constant +from nvflare.app_common.xgb.runners.xgb_runner import XGBRunner +from nvflare.app_common.xgb.sender import Sender +from nvflare.fuel.utils.validation_utils import check_non_negative_int, check_object_type, check_positive_int + + +class XGBAdaptor(ABC, FLComponent): + """ + XGBAdaptors are used to integrate FLARE with XGBoost Target (Server or Client) in run time. + + For example, an XGB server could be run as a gRPC server process, or be run as part of the FLARE's FL server + process. Similarly, an XGB client could be run as a gRPC client process, or be run as part of the + FLARE's FL client process. + + Each type of XGB Target requires an appropriate adaptor to integrate it with FLARE's XGB Controller or Executor. + + The XGBAdaptor class defines commonly required methods for all adaptor implementations. + """ + + def __init__(self): + FLComponent.__init__(self) + self.abort_signal = None + self.xgb_runner = None + + def set_runner(self, runner: XGBRunner): + """Set the XGB Runner that will be used to run XGB processing logic. + Note that the adaptor is only responsible for starting the runner appropriately (in a thread or in a + separate process). + + Args: + runner: the runner to be set + + Returns: None + + """ + if not isinstance(runner, XGBRunner): + raise TypeError(f"runner must be XGBRunner but got {type(runner)}") + self.xgb_runner = runner + + def set_abort_signal(self, abort_signal: Signal): + """Called by XGB Controller/Executor to set the abort_signal. + + The abort_signal is assigned by FLARE's XGB Controller/Executor. It is used by the Controller/Executor + to tell the adaptor that the job has been aborted. + + Args: + abort_signal: the abort signal assigned by the caller. + + Returns: None + + """ + check_object_type("abort_signal", abort_signal, Signal) + self.abort_signal = abort_signal + + def initialize(self, fl_ctx: FLContext): + """Called by the Controller/Executor to initialize the adaptor. + + Args: + fl_ctx: the FL context + + Returns: None + + """ + pass + + @abstractmethod + def start(self, fl_ctx: FLContext): + """Called by XGB Controller/Executor to start the target. + If any error occurs when starting the target, this method should raise an exception. + + Args: + fl_ctx: the FL context. + + Returns: None + + """ + pass + + @abstractmethod + def stop(self, fl_ctx: FLContext): + """Called by XGB Controller/Executor to stop the target. + If any error occurs when stopping the target, this method should raise an exception. + + Args: + fl_ctx: the FL context. + + Returns: None + + """ + pass + + @abstractmethod + def configure(self, config: dict, fl_ctx: FLContext): + """Called by XGB Controller/Executor to configure the adaptor. + If any error occurs, this method should raise an exception. + + Args: + config: config data + fl_ctx: the FL context + + Returns: None + + """ + pass + + @abstractmethod + def _is_stopped(self) -> (bool, int): + """Called by the adaptor's monitor to know whether the target is stopped. + Note that this method is not called by XGB Controller/Executor. + + Returns: a tuple of: whether the target is stopped, and return code (if stopped) + + Note that a non-zero return code is considered abnormal completion of the target. + + """ + pass + + def _monitor(self, fl_ctx: FLContext, target_stopped_cb): + while True: + if self.abort_signal.triggered: + # asked to abort + self.stop(fl_ctx) + return + + stopped, rc = self._is_stopped() + if stopped: + # target already stopped - notify the caller + target_stopped_cb(rc, fl_ctx) + return + + time.sleep(0.1) + + def monitor_target(self, fl_ctx: FLContext, target_stopped_cb): + """Called by XGB Controller/Executor to monitor the health of the target. + + The monitor periodically checks the abort signal. Once set, it calls the adaptor's stop() method + to stop the running of the target. + + The monitor also periodically checks whether the target is already stopped (by calling the is_stopped + method). If the target is stopped, the monitor will call the specified target_stopped_cb. + + Args: + fl_ctx: FL context + target_stopped_cb: the callback function to be called when the target is stopped. + + Returns: None + + """ + if not callable(target_stopped_cb): + raise RuntimeError(f"target_stopped_cb must be callable but got {type(target_stopped_cb)}") + + # start the monitor in a separate daemon thread! + t = threading.Thread(target=self._monitor, args=(fl_ctx, target_stopped_cb), daemon=True) + t.start() + + +class XGBServerAdaptor(XGBAdaptor): + """ + XGBServerAdaptor specifies commonly required methods for server adaptor implementations. + """ + + def __init__(self): + XGBAdaptor.__init__(self) + self.world_size = None + + def configure(self, config: dict, fl_ctx: FLContext): + """Called by XGB Controller to configure the target. + + The world_size is a required config parameter. + + Args: + config: config data + fl_ctx: FL context + + Returns: None + + """ + ws = config.get(Constant.CONF_KEY_WORLD_SIZE) + if not ws: + raise RuntimeError("world_size is not configured") + + check_positive_int(Constant.CONF_KEY_WORLD_SIZE, ws) + self.world_size = ws + + @abstractmethod + def all_gather(self, rank: int, seq: int, send_buf: bytes, fl_ctx: FLContext) -> bytes: + """Called by the XGB Controller to perform Allgather operation, per XGBoost spec. + + Args: + rank: rank of the calling client + seq: sequence number of the request + send_buf: operation input data + fl_ctx: FL context + + Returns: operation result + + """ + pass + + @abstractmethod + def all_gather_v(self, rank: int, seq: int, send_buf: bytes, fl_ctx: FLContext) -> bytes: + """Called by the XGB Controller to perform AllgatherV operation, per XGBoost spec. + + Args: + rank: rank of the calling client + seq: sequence number of the request + send_buf: input data + fl_ctx: FL context + + Returns: operation result + + """ + pass + + @abstractmethod + def all_reduce( + self, + rank: int, + seq: int, + data_type: int, + reduce_op: int, + send_buf: bytes, + fl_ctx: FLContext, + ) -> bytes: + """Called by the XGB Controller to perform Allreduce operation, per XGBoost spec. + + Args: + rank: rank of the calling client + seq: sequence number of the request + data_type: data type of the input + reduce_op: reduce operation to be performed + send_buf: input data + fl_ctx: FL context + + Returns: operation result + + """ + pass + + @abstractmethod + def broadcast(self, rank: int, seq: int, root: int, send_buf: bytes, fl_ctx: FLContext) -> bytes: + """Called by the XGB Controller to perform Broadcast operation, per XGBoost spec. + + Args: + rank: rank of the calling client + seq: sequence number of the request + root: root rank of the broadcast + send_buf: input data + fl_ctx: FL context + + Returns: operation result + + """ + pass + + +class XGBClientAdaptor(XGBAdaptor): + """ + XGBClientAdaptor specifies commonly required methods for client adaptor implementations. + """ + + def __init__(self): + """Constructor of XGBClientAdaptor""" + XGBAdaptor.__init__(self) + self.engine = None + self.sender = None + self.stopped = False + self.rank = None + self.num_rounds = None + self.world_size = None + + def set_sender(self, sender: Sender): + """Set the sender to be used to send XGB operation requests to the server. + + Args: + sender: the sender to be set + + Returns: None + + """ + if not isinstance(sender, Sender): + raise TypeError(f"sender must be Sender but got {type(sender)}") + self.sender = sender + + def configure(self, config: dict, fl_ctx: FLContext): + """Called by XGB Executor to configure the target. + + The rank, world size, and number of rounds are required config parameters. + + Args: + config: config data + fl_ctx: FL context + + Returns: None + + """ + ws = config.get(Constant.CONF_KEY_WORLD_SIZE) + if not ws: + raise RuntimeError("world_size is not configured") + + check_positive_int(Constant.CONF_KEY_WORLD_SIZE, ws) + self.world_size = ws + + rank = config.get(Constant.CONF_KEY_RANK) + if rank is None: + raise RuntimeError("rank is not configured") + + check_non_negative_int(Constant.CONF_KEY_RANK, rank) + self.rank = rank + + num_rounds = config.get(Constant.CONF_KEY_NUM_ROUNDS) + if num_rounds is None: + raise RuntimeError("num_rounds is not configured") + + check_positive_int(Constant.CONF_KEY_NUM_ROUNDS, num_rounds) + self.num_rounds = num_rounds + + def _send_request(self, op: str, req: Shareable) -> bytes: + """Send XGB operation request to the FL server via FLARE message. + + Args: + op: the XGB operation + req: operation data + + Returns: operation result + + """ + reply = self.sender.send_to_server(op, req, self.abort_signal) + if isinstance(reply, Shareable): + rcv_buf = reply.get(Constant.PARAM_KEY_RCV_BUF) + if not isinstance(rcv_buf, bytes): + raise RuntimeError(f"invalid rcv_buf for {op=}: expect bytes but got {type(rcv_buf)}") + return rcv_buf + else: + raise RuntimeError(f"invalid reply for op {op}: expect Shareable but got {type(reply)}") + + def _send_all_gather(self, rank: int, seq: int, send_buf: bytes) -> bytes: + """This method is called by a concrete client adaptor to send Allgather operation to the server. + + Args: + rank: rank of the client + seq: sequence number of the request + send_buf: input data + + Returns: operation result + + """ + req = Shareable() + req[Constant.PARAM_KEY_RANK] = rank + req[Constant.PARAM_KEY_SEQ] = seq + req[Constant.PARAM_KEY_SEND_BUF] = send_buf + return self._send_request(Constant.OP_ALL_GATHER, req) + + def _send_all_gather_v(self, rank: int, seq: int, send_buf: bytes) -> bytes: + """This method is called by a concrete client adaptor to send AllgatherV operation to the server. + + Args: + rank: rank of the client + seq: sequence number of the request + send_buf: operation input + + Returns: operation result + + """ + req = Shareable() + req[Constant.PARAM_KEY_RANK] = rank + req[Constant.PARAM_KEY_SEQ] = seq + req[Constant.PARAM_KEY_SEND_BUF] = send_buf + return self._send_request(Constant.OP_ALL_GATHER_V, req) + + def _send_all_reduce(self, rank: int, seq: int, data_type: int, reduce_op: int, send_buf: bytes) -> bytes: + """This method is called by a concrete client adaptor to send Allreduce operation to the server. + + Args: + rank: rank of the client + seq: sequence number of the request + data_type: data type of the input + reduce_op: reduce operation to be performed + send_buf: operation input + + Returns: operation result + + """ + req = Shareable() + req[Constant.PARAM_KEY_RANK] = rank + req[Constant.PARAM_KEY_SEQ] = seq + req[Constant.PARAM_KEY_DATA_TYPE] = data_type + req[Constant.PARAM_KEY_REDUCE_OP] = reduce_op + req[Constant.PARAM_KEY_SEND_BUF] = send_buf + return self._send_request(Constant.OP_ALL_REDUCE, req) + + def _send_broadcast(self, rank: int, seq: int, root: int, send_buf: bytes) -> bytes: + """This method is called by a concrete client adaptor to send Broadcast operation to the server. + + Args: + rank: rank of the client + seq: sequence number of the request + root: root rank of the broadcast + send_buf: operation input + + Returns: operation result + + """ + req = Shareable() + req[Constant.PARAM_KEY_RANK] = rank + req[Constant.PARAM_KEY_SEQ] = seq + req[Constant.PARAM_KEY_ROOT] = root + req[Constant.PARAM_KEY_SEND_BUF] = send_buf + return self._send_request(Constant.OP_BROADCAST, req) diff --git a/nvflare/app_common/xgb/adaptors/grpc_client_adaptor.py b/nvflare/app_common/xgb/adaptors/grpc_client_adaptor.py new file mode 100644 index 0000000000..e2f82b7483 --- /dev/null +++ b/nvflare/app_common/xgb/adaptors/grpc_client_adaptor.py @@ -0,0 +1,253 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import multiprocessing +import threading + +import nvflare.app_common.xgb.proto.federated_pb2 as pb2 +from nvflare.apis.fl_constant import FLContextKey +from nvflare.apis.fl_context import FLContext +from nvflare.app_common.xgb.adaptors.adaptor import XGBClientAdaptor +from nvflare.app_common.xgb.defs import Constant +from nvflare.app_common.xgb.grpc_server import GrpcServer +from nvflare.app_common.xgb.proto.federated_pb2_grpc import FederatedServicer +from nvflare.fuel.f3.drivers.net_utils import get_open_tcp_port +from nvflare.security.logging import secure_format_exception, secure_log_traceback + + +class _ClientStarter: + """This small class is used to start XGB client runner. It is used when running the runner in a thread + or in a separate process. + + """ + + def __init__(self, runner): + self.xgb_runner = runner + self.error = None + self.started = True + self.stopped = False + + def start(self, ctx: dict): + """Start the runner and wait for it to finish. + + Args: + ctx: + + Returns: + + """ + try: + self.xgb_runner.run(ctx) + self.stopped = True + except Exception as e: + secure_log_traceback() + self.error = f"Exception happens when running xgb train: {secure_format_exception(e)}" + self.started = False + + +class GrpcClientAdaptor(XGBClientAdaptor, FederatedServicer): + def __init__( + self, + int_server_grpc_options=None, + in_process=True, + ): + XGBClientAdaptor.__init__(self) + self.int_server_grpc_options = int_server_grpc_options + self.in_process = in_process + self.internal_xgb_server = None + self.stopped = False + self.internal_server_addr = None + self._training_stopped = False + self._client_name = None + self._app_dir = None + self._workspace = None + self._run_dir = None + self._process = None + self._starter = None + + def initialize(self, fl_ctx: FLContext): + self._client_name = fl_ctx.get_identity_name() + engine = fl_ctx.get_engine() + ws = engine.get_workspace() + self._app_dir = ws.get_app_dir(fl_ctx.get_job_id()) + self._workspace = fl_ctx.get_prop(FLContextKey.WORKSPACE_OBJECT) + run_number = fl_ctx.get_prop(FLContextKey.CURRENT_RUN) + self._run_dir = self._workspace.get_run_dir(run_number) + + def _start_client(self, server_addr: str): + """Start the XGB client runner in a separate thread or separate process based on config. + Note that when starting runner in a separate process, we must not call a method defined in this + class since the self object contains a sender that contains a Core Cell which cannot be sent to + the new process. Instead, we use a small _ClientStarter object to run the process. + + Args: + server_addr: the internal gRPC server address that the XGB client will connect to + + Returns: None + + """ + ctx = { + Constant.RUNNER_CTX_WORLD_SIZE: self.world_size, + Constant.RUNNER_CTX_CLIENT_NAME: self._client_name, + Constant.RUNNER_CTX_SERVER_ADDR: server_addr, + Constant.RUNNER_CTX_RANK: self.rank, + Constant.RUNNER_CTX_NUM_ROUNDS: self.num_rounds, + Constant.RUNNER_CTX_MODEL_DIR: self._run_dir, + Constant.RUNNER_CTX_TB_DIR: self._app_dir, + } + starter = _ClientStarter(self.xgb_runner) + if self.in_process: + self.logger.info("starting XGB client in another thread") + t = threading.Thread( + target=starter.start, + args=(ctx,), + daemon=True, + name="xgb_client_thread_runner", + ) + t.start() + if not starter.started: + self.logger.error(f"cannot start XGB client: {starter.error}") + raise RuntimeError(starter.error) + self._starter = starter + else: + # start as a separate local process + self.logger.info("starting XGB client in another process") + self._process = multiprocessing.Process( + target=starter.start, + args=(ctx,), + daemon=True, + name="xgb_client_process_runner", + ) + self._process.start() + + def _stop_client(self): + self._training_stopped = True + if self.in_process: + if self.xgb_runner: + self.xgb_runner.stop() + else: + if self._process: + self._process.kill() + + def _is_stopped(self) -> (bool, int): + if self.in_process: + if self._starter: + if self._starter.stopped: + return True, 0 + + if self._training_stopped: + return True, 0 + + if self.xgb_runner: + return self.xgb_runner.is_stopped() + else: + return True, 0 + else: + if self._process: + assert isinstance(self._process, multiprocessing.Process) + ec = self._process.exitcode + if ec is None: + return False, 0 + else: + return True, ec + else: + return True, 0 + + def start(self, fl_ctx: FLContext): + if self.rank is None: + raise RuntimeError("cannot start - my rank is not set") + + if not self.num_rounds: + raise RuntimeError("cannot start - num_rounds is not set") + + # dynamically determine address on localhost + port = get_open_tcp_port(resources={}) + if not port: + raise RuntimeError("failed to get a port for XGB server") + self.internal_server_addr = f"127.0.0.1:{port}" + self.logger.info(f"Start internal server at {self.internal_server_addr}") + self.internal_xgb_server = GrpcServer(self.internal_server_addr, 10, self.int_server_grpc_options, self) + self.internal_xgb_server.start(no_blocking=True) + self.logger.info(f"Started internal server at {self.internal_server_addr}") + self._start_client(self.internal_server_addr) + self.logger.info("Started external XGB Client") + + def stop(self, fl_ctx: FLContext): + if self.stopped: + return + + self.stopped = True + self._stop_client() + + if self.internal_xgb_server: + self.logger.info("Stop internal XGB Server") + self.internal_xgb_server.shutdown() + + def _abort(self, reason: str): + # stop the gRPC XGB client (the target) + self.abort_signal.trigger(True) + + # abort the FL client + with self.engine.new_context() as fl_ctx: + self.system_panic(reason, fl_ctx) + + def Allgather(self, request: pb2.AllgatherRequest, context): + try: + rcv_buf = self._send_all_gather( + rank=request.rank, + seq=request.sequence_number, + send_buf=request.send_buffer, + ) + return pb2.AllgatherReply(receive_buffer=rcv_buf) + except Exception as ex: + self._abort(reason=f"send_all_gather exception: {secure_format_exception(ex)}") + return None + + def AllgatherV(self, request: pb2.AllgatherVRequest, context): + try: + rcv_buf = self._send_all_gather_v( + rank=request.rank, + seq=request.sequence_number, + send_buf=request.send_buffer, + ) + return pb2.AllgatherVReply(receive_buffer=rcv_buf) + except Exception as ex: + self._abort(reason=f"send_all_gather_v exception: {secure_format_exception(ex)}") + return None + + def Allreduce(self, request: pb2.AllreduceRequest, context): + try: + rcv_buf = self._send_all_reduce( + rank=request.rank, + seq=request.sequence_number, + data_type=request.data_type, + reduce_op=request.reduce_operation, + send_buf=request.send_buffer, + ) + return pb2.AllreduceReply(receive_buffer=rcv_buf) + except Exception as ex: + self._abort(reason=f"send_all_reduce exception: {secure_format_exception(ex)}") + return None + + def Broadcast(self, request: pb2.BroadcastRequest, context): + try: + rcv_buf = self._send_broadcast( + rank=request.rank, + seq=request.sequence_number, + root=request.root, + send_buf=request.send_buffer, + ) + return pb2.BroadcastReply(receive_buffer=rcv_buf) + except Exception as ex: + self._abort(reason=f"send_broadcast exception: {secure_format_exception(ex)}") + return None diff --git a/nvflare/app_common/xgb/adaptors/grpc_server_adaptor.py b/nvflare/app_common/xgb/adaptors/grpc_server_adaptor.py new file mode 100644 index 0000000000..540c164745 --- /dev/null +++ b/nvflare/app_common/xgb/adaptors/grpc_server_adaptor.py @@ -0,0 +1,158 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import multiprocessing +import threading + +import nvflare.app_common.xgb.proto.federated_pb2 as pb2 +from nvflare.apis.fl_context import FLContext +from nvflare.app_common.xgb.adaptors.adaptor import XGBServerAdaptor +from nvflare.app_common.xgb.defs import Constant +from nvflare.app_common.xgb.grpc_client import GrpcClient +from nvflare.fuel.f3.drivers.net_utils import get_open_tcp_port +from nvflare.security.logging import secure_format_exception + + +class GrpcServerAdaptor(XGBServerAdaptor): + def __init__( + self, + int_client_grpc_options=None, + xgb_server_ready_timeout=Constant.XGB_SERVER_READY_TIMEOUT, + in_process=True, + ): + XGBServerAdaptor.__init__(self) + self.int_client_grpc_options = int_client_grpc_options + self.xgb_server_ready_timeout = xgb_server_ready_timeout + self.in_process = in_process + self.internal_xgb_client = None + self._process = None + self._server_stopped = False + + def _try_start_server(self, addr: str, port: int, world_size: int): + ctx = { + Constant.RUNNER_CTX_SERVER_ADDR: addr, + Constant.RUNNER_CTX_WORLD_SIZE: world_size, + Constant.RUNNER_CTX_PORT: port, + } + try: + self.xgb_runner.run(ctx) + except Exception as ex: + self.logger.error(f"Exception running xgb_runner {ctx=}: {secure_format_exception(ex)}") + raise ex + + def _start_server(self, addr: str, port: int, world_size: int): + if self.in_process: + self.logger.info("starting XGB server in another thread") + t = threading.Thread( + name="xgb_server_thread", target=self._try_start_server, args=(addr, port, world_size), daemon=True + ) + t.start() + else: + self.logger.info("starting XGB server in another process") + self._process = multiprocessing.Process( + name="xgb_server_process", target=self._try_start_server, args=(addr, port, world_size), daemon=True + ) + self._process.start() + + def _stop_server(self): + self._server_stopped = True + if self.in_process: + if self.xgb_runner: + self.xgb_runner.stop() + else: + if self._process: + self._process.kill() + self._process = None + + def _is_stopped(self) -> (bool, int): + if self._server_stopped: + return True, 0 + + if self.in_process: + if self.xgb_runner: + return self.xgb_runner.is_stopped() + else: + return True, 0 + else: + if self._process: + assert isinstance(self._process, multiprocessing.Process) + ec = self._process.exitcode + if ec is None: + return False, 0 + else: + return True, ec + else: + return True, 0 + + def start(self, fl_ctx: FLContext): + # we dynamically create server address on localhost + port = get_open_tcp_port(resources={}) + if not port: + raise RuntimeError("failed to get a port for XGB server") + + server_addr = f"127.0.0.1:{port}" + self._start_server(server_addr, port, self.world_size) + + # start XGB client + self.internal_xgb_client = GrpcClient(server_addr, self.int_client_grpc_options) + self.internal_xgb_client.start(ready_timeout=self.xgb_server_ready_timeout) + + def stop(self, fl_ctx: FLContext): + client = self.internal_xgb_client + self.internal_xgb_client = None + if client: + self.log_info(fl_ctx, "Stopping internal XGB client") + client.stop() + self._stop_server() + + def all_gather(self, rank: int, seq: int, send_buf: bytes, fl_ctx: FLContext) -> bytes: + result = self.internal_xgb_client.send_allgather(seq_num=seq, rank=rank, data=send_buf) + if isinstance(result, pb2.AllgatherReply): + return result.receive_buffer + else: + raise RuntimeError(f"bad result from XGB server: expect AllgatherReply but got {type(result)}") + + def all_gather_v(self, rank: int, seq: int, send_buf: bytes, fl_ctx: FLContext) -> bytes: + result = self.internal_xgb_client.send_allgatherv(seq_num=seq, rank=rank, data=send_buf) + if isinstance(result, pb2.AllgatherVReply): + return result.receive_buffer + else: + raise RuntimeError(f"bad result from XGB server: expect AllgatherVReply but got {type(result)}") + + def all_reduce( + self, + rank: int, + seq: int, + data_type: int, + reduce_op: int, + send_buf: bytes, + fl_ctx: FLContext, + ) -> bytes: + result = self.internal_xgb_client.send_allreduce( + seq_num=seq, + rank=rank, + data=send_buf, + data_type=data_type, + reduce_op=reduce_op, + ) + if isinstance(result, pb2.AllreduceReply): + return result.receive_buffer + else: + raise RuntimeError(f"bad result from XGB server: expect AllreduceReply but got {type(result)}") + + def broadcast(self, rank: int, seq: int, root: int, send_buf: bytes, fl_ctx: FLContext) -> bytes: + result = self.internal_xgb_client.send_broadcast(seq_num=seq, rank=rank, data=send_buf, root=root) + if isinstance(result, pb2.BroadcastReply): + return result.receive_buffer + else: + raise RuntimeError(f"bad result from XGB server: expect BroadcastReply but got {type(result)}") diff --git a/nvflare/app_common/xgb/controller.py b/nvflare/app_common/xgb/controller.py new file mode 100644 index 0000000000..83bd6f90a1 --- /dev/null +++ b/nvflare/app_common/xgb/controller.py @@ -0,0 +1,609 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import threading +import time + +from nvflare.apis.client import Client +from nvflare.apis.fl_context import FLContext +from nvflare.apis.impl.controller import ClientTask, Controller, Task +from nvflare.apis.shareable import ReturnCode, Shareable, make_reply +from nvflare.apis.signal import Signal +from nvflare.app_common.xgb.adaptors.adaptor import XGBServerAdaptor +from nvflare.fuel.utils.validation_utils import check_number_range, check_object_type, check_positive_number, check_str +from nvflare.security.logging import secure_format_exception + +from .defs import Constant + + +class ClientStatus: + """ + Objects of this class keep processing status of each FL client during job execution. + """ + + def __init__(self): + # Set when the client's config reply is received and the reply return code is OK. + # If the client failed to reply or the return code is not OK, this value is not set. + self.configured_time = None + + # Set when the client's start reply is received and the reply return code is OK. + # If the client failed to reply or the return code is not OK, this value is not set. + self.started_time = None + + # operation of the last XGB request from this client + self.last_op = None + + # time of the last XGB op request from this client + self.last_op_time = time.time() + + # whether the XGB process is done on this client + self.xgb_done = False + + +class XGBController(Controller): + def __init__( + self, + adaptor_component_id: str, + num_rounds: int, + configure_task_name=Constant.CONFIG_TASK_NAME, + configure_task_timeout=Constant.CONFIG_TASK_TIMEOUT, + start_task_name=Constant.START_TASK_NAME, + start_task_timeout=Constant.START_TASK_TIMEOUT, + job_status_check_interval: float = Constant.JOB_STATUS_CHECK_INTERVAL, + max_client_op_interval: float = Constant.MAX_CLIENT_OP_INTERVAL, + progress_timeout: float = Constant.WORKFLOW_PROGRESS_TIMEOUT, + client_ranks=None, + ): + """ + Constructor + + Args: + adaptor_component_id - the component ID of server target adaptor + num_rounds - number of rounds + configure_task_name - name of the config task + configure_task_timeout - time to wait for clients’ responses to the config task before timeout. + start_task_name - name of the start task + start_task_timeout - time to wait for clients’ responses to the start task before timeout. + job_status_check_interval - how often to check client statuses of the job + max_client_op_interval - max amount of time allowed between XGB ops from a client + progress_timeout- the maximum amount of time allowed for the workflow to not make any progress. + In other words, at least one participating client must have made progress during this time. + Otherwise, the workflow will be considered to be in trouble and the job will be aborted. + client_ranks: client rank assignments. + If specified, must be a dict of client_name => rank. + If not specified, client ranks will be randomly assigned. + No matter how assigned, ranks must be consecutive integers, starting from 0. + """ + Controller.__init__(self) + self.adaptor_component_id = adaptor_component_id + self.num_rounds = num_rounds + self.configure_task_name = configure_task_name + self.start_task_name = start_task_name + self.start_task_timeout = start_task_timeout + self.configure_task_timeout = configure_task_timeout + self.max_client_op_interval = max_client_op_interval + self.progress_timeout = progress_timeout + self.job_status_check_interval = job_status_check_interval + self.client_ranks = client_ranks # client rank assignments + + self.adaptor = None + self.participating_clients = None + self.status_lock = threading.Lock() + self.client_statuses = {} # client name => ClientStatus + self.abort_signal = None + + check_str("adaptor_component_id", adaptor_component_id) + check_number_range("configure_task_timeout", configure_task_timeout, min_value=1) + check_number_range("start_task_timeout", start_task_timeout, min_value=1) + check_positive_number("job_status_check_interval", job_status_check_interval) + check_positive_number("num_rounds", num_rounds) + check_number_range("max_client_op_interval", max_client_op_interval, min_value=10.0) + check_number_range("progress_timeout", progress_timeout, min_value=5.0) + if client_ranks: + check_object_type("client_ranks", client_ranks, dict) + + # set up operation handlers + self.op_table = { + Constant.OP_ALL_GATHER: self._process_all_gather, + Constant.OP_ALL_GATHER_V: self._process_all_gather_v, + Constant.OP_ALL_REDUCE: self._process_all_reduce, + Constant.OP_BROADCAST: self._process_broadcast, + } + + def get_adaptor(self, fl_ctx: FLContext): + engine = fl_ctx.get_engine() + return engine.get_component(self.adaptor_component_id) + + def start_controller(self, fl_ctx: FLContext): + all_clients = self._engine.get_clients() + self.participating_clients = [t.name for t in all_clients] + + for c in self.participating_clients: + self.client_statuses[c] = ClientStatus() + + adaptor = self.get_adaptor(fl_ctx) + if not adaptor: + self.system_panic(f"cannot get component for {self.adaptor_component_id}", fl_ctx) + return None + + if not isinstance(adaptor, XGBServerAdaptor): + self.system_panic( + f"invalid component '{self.adaptor_component_id}': expect XGBServerBridge but got {type(adaptor)}", + fl_ctx, + ) + return None + + adaptor.initialize(fl_ctx) + self.adaptor = adaptor + + engine = fl_ctx.get_engine() + engine.register_aux_message_handler( + topic=Constant.TOPIC_XGB_REQUEST, + message_handle_func=self._process_xgb_request, + ) + engine.register_aux_message_handler( + topic=Constant.TOPIC_CLIENT_DONE, + message_handle_func=self._process_client_done, + ) + + def _trigger_stop(self, fl_ctx: FLContext, error=None): + # first trigger the abort_signal to tell all components (mainly the controller's control_flow and adaptor) + # that check this signal to abort. + if self.abort_signal: + self.abort_signal.trigger(value=True) + + # if there is error, call system_panic to terminate the job with proper status. + # if no error, the job will end normally. + if error: + self.system_panic(reason=error, fl_ctx=fl_ctx) + + def _is_stopped(self): + # check whether the abort signal is triggered + return self.abort_signal and self.abort_signal.triggered + + def _update_client_status(self, fl_ctx: FLContext, op=None, client_done=False): + """Update the status of the requesting client. + + Args: + fl_ctx: FL context + op: the XGB operation requested + client_done: whether the client is done + + Returns: None + + """ + with self.status_lock: + peer_ctx = fl_ctx.get_peer_context() + if not peer_ctx: + self.log_error(fl_ctx, "missing peer_ctx from fl_ctx") + return + if not isinstance(peer_ctx, FLContext): + self.log_error(fl_ctx, f"expect peer_ctx to be FLContext but got {type(peer_ctx)}") + return + client_name = peer_ctx.get_identity_name() + if not client_name: + self.log_error(fl_ctx, "missing identity from peer_ctx") + return + status = self.client_statuses.get(client_name) + if not status: + self.log_error(fl_ctx, f"no status record for client {client_name}") + assert isinstance(status, ClientStatus) + if op: + status.last_op = op + if client_done: + status.xgb_done = client_done + status.last_op_time = time.time() + + def _process_client_done(self, topic: str, request: Shareable, fl_ctx: FLContext) -> Shareable: + """Process the ClientDone report for a client + + Args: + topic: topic of the message + request: request to be processed + fl_ctx: the FL context + + Returns: reply to the client + + """ + exit_code = request.get(Constant.MSG_KEY_EXIT_CODE) + + # TBD: should we check the exit_code and determine job status? + # Problem is that even if the exit_code is not 0, we can't say the job failed. + if exit_code == 0: + self.log_info(fl_ctx, f"XGB client is done with exit code {exit_code}") + else: + self.log_warning(fl_ctx, f"XGB client is done with exit code {exit_code}") + + self._update_client_status(fl_ctx, client_done=True) + return make_reply(ReturnCode.OK) + + def _process_all_gather(self, request: Shareable, fl_ctx: FLContext) -> Shareable: + """This is the op handler for Allgather. + + Args: + request: the request containing op params + fl_ctx: FL context + + Returns: a Shareable containing operation result + + """ + rank = request.get(Constant.PARAM_KEY_RANK) + seq = request.get(Constant.PARAM_KEY_SEQ) + send_buf = request.get(Constant.PARAM_KEY_SEND_BUF) + rcv_buf = self.adaptor.all_gather(rank, seq, send_buf, fl_ctx) + reply = Shareable() + reply[Constant.PARAM_KEY_RCV_BUF] = rcv_buf + return reply + + def _process_all_gather_v(self, request: Shareable, fl_ctx: FLContext) -> Shareable: + """This is the op handler for AllgatherV. + + Args: + request: the request containing op params + fl_ctx: FL context + + Returns: a Shareable containing operation result + + """ + rank = request.get(Constant.PARAM_KEY_RANK) + seq = request.get(Constant.PARAM_KEY_SEQ) + send_buf = request.get(Constant.PARAM_KEY_SEND_BUF) + rcv_buf = self.adaptor.all_gather_v(rank, seq, send_buf, fl_ctx) + reply = Shareable() + reply[Constant.PARAM_KEY_RCV_BUF] = rcv_buf + return reply + + def _process_all_reduce(self, request: Shareable, fl_ctx: FLContext) -> Shareable: + """This is the op handler for Allreduce. + + Args: + request: the request containing op params + fl_ctx: FL context + + Returns: a Shareable containing operation result + + """ + rank = request.get(Constant.PARAM_KEY_RANK) + seq = request.get(Constant.PARAM_KEY_SEQ) + send_buf = request.get(Constant.PARAM_KEY_SEND_BUF) + data_type = request.get(Constant.PARAM_KEY_DATA_TYPE) + reduce_op = request.get(Constant.PARAM_KEY_REDUCE_OP) + assert isinstance(self.adaptor, XGBServerAdaptor) + rcv_buf = self.adaptor.all_reduce(rank, seq, data_type, reduce_op, send_buf, fl_ctx) + reply = Shareable() + reply[Constant.PARAM_KEY_RCV_BUF] = rcv_buf + return reply + + def _process_broadcast(self, request: Shareable, fl_ctx: FLContext) -> Shareable: + """This is the op handler for Broadcast. + + Args: + request: the request containing op params + fl_ctx: FL context + + Returns: a Shareable containing operation result + + """ + rank = request.get(Constant.PARAM_KEY_RANK) + seq = request.get(Constant.PARAM_KEY_SEQ) + send_buf = request.get(Constant.PARAM_KEY_SEND_BUF) + root = request.get(Constant.PARAM_KEY_ROOT) + assert isinstance(self.adaptor, XGBServerAdaptor) + rcv_buf = self.adaptor.broadcast(rank, seq, root, send_buf, fl_ctx) + reply = Shareable() + reply[Constant.PARAM_KEY_RCV_BUF] = rcv_buf + return reply + + def _process_xgb_request(self, topic: str, request: Shareable, fl_ctx: FLContext) -> Shareable: + op = request.get_header(Constant.MSG_KEY_XGB_OP) + if self._is_stopped(): + self.log_error(fl_ctx, f"dropped XGB request '{op}' since server is already stopped") + return make_reply(ReturnCode.SERVICE_UNAVAILABLE) + + # since XGB protocol is very strict, we'll stop the control flow when any error occurs + bad_req_error = "bad XGB request" + process_error = "XGB request process error" + if not op: + self.log_error(fl_ctx, "missing op from XGB request") + self._trigger_stop(fl_ctx, bad_req_error) + return make_reply(ReturnCode.BAD_REQUEST_DATA) + + # find and call the op handlers + process_f = self.op_table.get(op) + if process_f is None: + self.log_error(fl_ctx, f"invalid op '{op}' from XGB request") + self._trigger_stop(fl_ctx, bad_req_error) + return make_reply(ReturnCode.BAD_REQUEST_DATA) + + self._update_client_status(fl_ctx, op=op) + + if not callable(process_f): + # impossible but we must declare process_f to be callable; otherwise PyCharm will complain about + # process_f(request, fl_ctx). + raise RuntimeError(f"op handler for {op} is not callable") + try: + reply = process_f(request, fl_ctx) + except Exception as ex: + self.log_exception(fl_ctx, f"exception processing {op}: {secure_format_exception(ex)}") + self._trigger_stop(fl_ctx, process_error) + return make_reply(ReturnCode.EXECUTION_EXCEPTION) + + self.log_info(fl_ctx, f"received reply for '{op}'") + reply.set_header(Constant.MSG_KEY_XGB_OP, op) + return reply + + def _configure_clients(self, abort_signal: Signal, fl_ctx: FLContext): + self.log_info(fl_ctx, f"Configuring clients {self.participating_clients}") + + shareable = Shareable() + + # compute client ranks + if not self.client_ranks: + # dynamically assign ranks, starting from 0 + # Assumption: all clients are used + clients = self.participating_clients + + # Sort by client name so rank is consistent + clients.sort() + self.client_ranks = {clients[i]: i for i in range(0, len(clients))} + else: + # validate ranks - ranks must be unique consecutive integers, starting from 0. + num_clients = len(self.participating_clients) + assigned_ranks = {} # rank => client + if len(self.client_ranks) != num_clients: + # either missing client or duplicate client + self.system_panic( + f"expecting rank assignments for {self.participating_clients} but got {self.client_ranks}", fl_ctx + ) + return False + + # all clients must have ranks + for c in self.participating_clients: + if c not in self.client_ranks: + self.system_panic(f"missing rank assignment for client '{c}'", fl_ctx) + return False + + # check each client's rank + for c, r in self.client_ranks.items(): + if not isinstance(r, int): + self.system_panic(f"bad rank assignment {r} for client '{c}': expect int but got {type(r)}", fl_ctx) + return False + + if r < 0 or r >= num_clients: + self.system_panic(f"bad rank assignment {r} for client '{c}': must be 0 to {num_clients-1}", fl_ctx) + return False + + assigned_client = assigned_ranks.get(r) + if assigned_client: + self.system_panic(f"rank {r} is assigned to both client '{c}' and '{assigned_client}'", fl_ctx) + return False + + assigned_ranks[r] = c + + shareable[Constant.CONF_KEY_CLIENT_RANKS] = self.client_ranks + shareable[Constant.CONF_KEY_NUM_ROUNDS] = self.num_rounds + + task = Task( + name=self.configure_task_name, + data=shareable, + timeout=self.configure_task_timeout, + result_received_cb=self._process_configure_reply, + ) + + self.log_info(fl_ctx, f"sending task {self.configure_task_name} to clients {self.participating_clients}") + start_time = time.time() + self.broadcast_and_wait( + task=task, + targets=self.participating_clients, + min_responses=len(self.participating_clients), + fl_ctx=fl_ctx, + abort_signal=abort_signal, + ) + + time_taken = time.time() - start_time + self.log_info(fl_ctx, f"client configuration took {time_taken} seconds") + + failed_clients = [] + for c, cs in self.client_statuses.items(): + assert isinstance(cs, ClientStatus) + if not cs.configured_time: + failed_clients.append(c) + + # if any client failed to configure, terminate the job + if failed_clients: + self.system_panic(f"failed to configure clients {failed_clients}", fl_ctx) + return False + + self.log_info(fl_ctx, f"successfully configured clients {self.participating_clients}") + return True + + def _start_clients(self, abort_signal: Signal, fl_ctx: FLContext): + self.log_info(fl_ctx, f"Starting clients {self.participating_clients}") + + task = Task( + name=self.start_task_name, + data=Shareable(), + timeout=self.start_task_timeout, + result_received_cb=self._process_start_reply, + ) + + self.log_info(fl_ctx, f"sending task {self.start_task_name} to clients {self.participating_clients}") + start_time = time.time() + self.broadcast_and_wait( + task=task, + targets=self.participating_clients, + min_responses=len(self.participating_clients), + fl_ctx=fl_ctx, + abort_signal=abort_signal, + ) + + time_taken = time.time() - start_time + self.log_info(fl_ctx, f"client starting took {time_taken} seconds") + + failed_clients = [] + for c, cs in self.client_statuses.items(): + assert isinstance(cs, ClientStatus) + if not cs.started_time: + failed_clients.append(c) + + # if any client failed to start, terminate the job + if failed_clients: + self.system_panic(f"failed to start clients {failed_clients}", fl_ctx) + return False + + self.log_info(fl_ctx, f"successfully started clients {self.participating_clients}") + return True + + def control_flow(self, abort_signal: Signal, fl_ctx: FLContext): + """ + This is the control flow of the XGB Controller. To ensure smooth XGB execution: + - ensure that all clients are online and ready to go before starting server + - ensure that server is started and ready to take requests before asking clients to start operation + - monitor the health of the clients + - if anything goes wrong, terminate the job + + Args: + abort_signal: abort signal that is used to notify components to abort + fl_ctx: FL context + + Returns: None + + """ + self.abort_signal = abort_signal + + # the adaptor uses the same abort signal! + self.adaptor.set_abort_signal(abort_signal) + + # wait for every client to become online and properly configured + self.log_info(fl_ctx, f"Waiting for clients to be ready: {self.participating_clients}") + + # configure all clients + if not self._configure_clients(abort_signal, fl_ctx): + self.system_panic("failed to configure all clients", fl_ctx) + return + + # start the server adaptor + try: + self.adaptor.configure({Constant.CONF_KEY_WORLD_SIZE: len(self.participating_clients)}, fl_ctx) + self.adaptor.start(fl_ctx) + except Exception as ex: + error = f"failed to start bridge: {secure_format_exception(ex)}" + self.log_error(fl_ctx, error) + self.system_panic(error, fl_ctx) + return + + self.adaptor.monitor_target(fl_ctx, self._xgb_server_stopped) + + # start all clients + if not self._start_clients(abort_signal, fl_ctx): + self.system_panic("failed to start all clients", fl_ctx) + return + + # monitor client health + # we periodically check job status until all clients are done or the system is stopped + self.log_info(fl_ctx, "Waiting for clients to finish ...") + while not self._is_stopped(): + done = self._check_job_status(fl_ctx) + if done: + break + time.sleep(self.job_status_check_interval) + + def _xgb_server_stopped(self, rc, fl_ctx: FLContext): + # This CB is called when XGB server target is stopped + error = None + if rc != 0: + self.log_error(fl_ctx, f"XGB Server stopped abnormally with code {rc}") + error = "XGB server abnormal stop" + + # the XGB server could stop at any moment, we trigger the abort_signal in case it is checked by any + # other components + self._trigger_stop(fl_ctx, error) + + def _process_configure_reply(self, client_task: ClientTask, fl_ctx: FLContext): + result = client_task.result + client_name = client_task.client.name + + rc = result.get_return_code() + if rc == ReturnCode.OK: + self.log_info(fl_ctx, f"successfully configured client {client_name}") + cs = self.client_statuses.get(client_name) + if cs: + assert isinstance(cs, ClientStatus) + cs.configured_time = time.time() + else: + self.log_error(fl_ctx, f"client {client_task.client.name} failed to configure: {rc}") + + def _process_start_reply(self, client_task: ClientTask, fl_ctx: FLContext): + result = client_task.result + client_name = client_task.client.name + + rc = result.get_return_code() + if rc == ReturnCode.OK: + self.log_info(fl_ctx, f"successfully started client {client_name}") + cs = self.client_statuses.get(client_name) + if cs: + assert isinstance(cs, ClientStatus) + cs.started_time = time.time() + else: + self.log_error(fl_ctx, f"client {client_name} failed to start") + + def _check_job_status(self, fl_ctx: FLContext) -> bool: + """Check job status and determine whether the job is done. + + Args: + fl_ctx: FL context + + Returns: whether the job is considered done. + + """ + now = time.time() + + # overall_last_progress_time is the latest time that any client made progress. + overall_last_progress_time = 0.0 + clients_done = 0 + for client_name, cs in self.client_statuses.items(): + assert isinstance(cs, ClientStatus) + + if cs.xgb_done: + self.log_info(fl_ctx, f"client {client_name} is Done") + clients_done += 1 + elif now - cs.last_op_time > self.max_client_op_interval: + self.system_panic( + f"client {client_name} didn't have any activity for {self.max_client_op_interval} seconds", + fl_ctx, + ) + return True + + if overall_last_progress_time < cs.last_op_time: + overall_last_progress_time = cs.last_op_time + + if clients_done == len(self.client_statuses): + # all clients are done - the job is considered done + return True + elif time.time() - overall_last_progress_time > self.progress_timeout: + # there has been no progress from any client for too long. + # this could be because the clients got stuck. + # consider the job done and abort the job. + self.system_panic(f"the job has no progress for {self.progress_timeout} seconds", fl_ctx) + return True + return False + + def process_result_of_unknown_task( + self, client: Client, task_name: str, client_task_id: str, result: Shareable, fl_ctx: FLContext + ): + self.log_warning(fl_ctx, f"ignored unknown task {task_name} from client {client.name}") + + def stop_controller(self, fl_ctx: FLContext): + if self.adaptor: + self.log_info(fl_ctx, "Stopping server bridge") + self.adaptor.stop(fl_ctx) diff --git a/nvflare/app_common/xgb/data_loader.py b/nvflare/app_common/xgb/data_loader.py new file mode 100644 index 0000000000..2fa8855c99 --- /dev/null +++ b/nvflare/app_common/xgb/data_loader.py @@ -0,0 +1,30 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from abc import ABC, abstractmethod +from typing import Tuple + +import xgboost as xgb + + +class XGBDataLoader(ABC): + @abstractmethod + def load_data(self, client_id: str) -> Tuple[xgb.core.DMatrix, xgb.core.DMatrix]: + """Loads data for xgboost. + + Returns: + A tuple of train_data, validation_data + """ + pass diff --git a/nvflare/app_common/xgb/defs.py b/nvflare/app_common/xgb/defs.py new file mode 100644 index 0000000000..c7997951f6 --- /dev/null +++ b/nvflare/app_common/xgb/defs.py @@ -0,0 +1,95 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from nvflare.fuel.f3.drivers.net_utils import MAX_FRAME_SIZE + + +class Constant: + + # task name defaults + CONFIG_TASK_NAME = "config" + START_TASK_NAME = "start" + + # keys of adaptor config parameters + CONF_KEY_CLIENT_RANKS = "client_ranks" + CONF_KEY_RANK = "rank" + CONF_KEY_WORLD_SIZE = "world_size" + CONF_KEY_NUM_ROUNDS = "num_rounds" + + # default component config values + CONFIG_TASK_TIMEOUT = 10 + START_TASK_TIMEOUT = 10 + XGB_SERVER_READY_TIMEOUT = 10.0 + + TASK_CHECK_INTERVAL = 0.5 + JOB_STATUS_CHECK_INTERVAL = 2.0 + MAX_CLIENT_OP_INTERVAL = 90.0 + WORKFLOW_PROGRESS_TIMEOUT = 3600.0 + + # message topics + TOPIC_XGB_REQUEST = "xgb.request" + TOPIC_XGB_REQUEST_CHECK = "xgb.req_check" + TOPIC_CLIENT_DONE = "xgb.client_done" + + # keys for Shareable between client and server + MSG_KEY_EXIT_CODE = "exit_code" + MSG_KEY_XGB_OP = "xgb.op" + MSG_KEY_XGB_REQ_ID = "xgb.req_id" + MSG_KEY_XGB_REQ_TRY_NUM = "xgb.req_try_num" + MSG_KEY_XGB_REQ_RECEIVED = "xgb.req_received" + + # XGB operation names + OP_ALL_GATHER = "all_gather" + OP_ALL_GATHER_V = "all_gather_v" + OP_ALL_REDUCE = "all_reduce" + OP_BROADCAST = "broadcast" + + # XGB operation codes + OPCODE_NONE = 0 + OPCODE_ALL_GATHER = 1 + OPCODE_ALL_GATHER_V = 2 + OPCODE_ALL_REDUCE = 3 + OPCODE_BROADCAST = 4 + OPCODE_DONE = 99 + + # XGB operation error codes + ERR_OP_MISMATCH = -1 + ERR_INVALID_RANK = -2 + ERR_NO_CLIENT_FOR_RANK = -3 + ERR_TARGET_ERROR = -4 + + # XGB operation parameter keys + PARAM_KEY_RANK = "xgb.rank" + PARAM_KEY_SEQ = "xgb.seq" + PARAM_KEY_SEND_BUF = "xgb.send_buf" + PARAM_KEY_DATA_TYPE = "xgb.data_type" + PARAM_KEY_REDUCE_OP = "xgb.reduce_op" + PARAM_KEY_ROOT = "xgb.root" + PARAM_KEY_RCV_BUF = "xgb.rcv_buf" + + RUNNER_CTX_SERVER_ADDR = "server_addr" + RUNNER_CTX_PORT = "port" + RUNNER_CTX_CLIENT_NAME = "client_name" + RUNNER_CTX_NUM_ROUNDS = "num_rounds" + RUNNER_CTX_WORLD_SIZE = "world_size" + RUNNER_CTX_RANK = "rank" + RUNNER_CTX_DATA_LOADER = "data_loader" + RUNNER_CTX_TB_DIR = "tb_dir" + RUNNER_CTX_MODEL_DIR = "model_dir" + + +GRPC_DEFAULT_OPTIONS = [ + ("grpc.max_send_message_length", MAX_FRAME_SIZE), + ("grpc.max_receive_message_length", MAX_FRAME_SIZE), +] diff --git a/nvflare/app_common/xgb/executor.py b/nvflare/app_common/xgb/executor.py new file mode 100644 index 0000000000..b647b3b308 --- /dev/null +++ b/nvflare/app_common/xgb/executor.py @@ -0,0 +1,171 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from nvflare.apis.event_type import EventType +from nvflare.apis.executor import Executor +from nvflare.apis.fl_constant import ReturnCode +from nvflare.apis.fl_context import FLContext +from nvflare.apis.shareable import Shareable, make_reply +from nvflare.apis.signal import Signal +from nvflare.app_common.xgb.adaptors.adaptor import XGBClientAdaptor +from nvflare.fuel.f3.cellnet.fqcn import FQCN +from nvflare.security.logging import secure_format_exception + +from .defs import Constant +from .sender import Sender + + +class XGBExecutor(Executor): + def __init__( + self, + adaptor_component_id: str, + configure_task_name=Constant.CONFIG_TASK_NAME, + start_task_name=Constant.START_TASK_NAME, + req_timeout=10.0, + ): + """Constructor + + Args: + adaptor_component_id: the component ID of client target adaptor + configure_task_name: name of the config task + start_task_name: name of the start task + """ + Executor.__init__(self) + self.adaptor_component_id = adaptor_component_id + self.req_timeout = req_timeout + self.configure_task_name = configure_task_name + self.start_task_name = start_task_name + self.adaptor = None + + # create the abort signal to be used for signaling the adaptor + self.abort_signal = Signal() + + def get_adaptor(self, fl_ctx: FLContext): + """Get adaptor to be used by this executor. + This is the default implementation that gets the adaptor based on configured adaptor_component_id. + A subclass of XGBExecutor may get adaptor in a different way. + + Args: + fl_ctx: the FL context + + Returns: a XGBClientAdaptor object + + """ + engine = fl_ctx.get_engine() + return engine.get_component(self.adaptor_component_id) + + def handle_event(self, event_type: str, fl_ctx: FLContext): + if event_type == EventType.START_RUN: + adaptor = self.get_adaptor(fl_ctx) + if not adaptor: + self.system_panic(f"cannot get component for {self.adaptor_component_id}", fl_ctx) + return + + if not isinstance(adaptor, XGBClientAdaptor): + self.system_panic( + f"invalid component '{self.adaptor_component_id}': expect XGBClientAdaptor but got {type(adaptor)}", + fl_ctx, + ) + return + + adaptor.set_abort_signal(self.abort_signal) + engine = fl_ctx.get_engine() + adaptor.set_sender(Sender(engine, self.req_timeout)) + adaptor.initialize(fl_ctx) + self.adaptor = adaptor + elif event_type == EventType.END_RUN: + self.abort_signal.trigger(True) + + def execute(self, task_name: str, shareable: Shareable, fl_ctx: FLContext, abort_signal: Signal) -> Shareable: + if task_name == self.configure_task_name: + # there are two important config params for the client: + # the rank assigned to the client; + # number of rounds for training. + ranks = shareable.get(Constant.CONF_KEY_CLIENT_RANKS) + if not ranks: + self.log_error(fl_ctx, f"missing {Constant.CONF_KEY_CLIENT_RANKS} from config") + return make_reply(ReturnCode.BAD_TASK_DATA) + + if not isinstance(ranks, dict): + self.log_error(fl_ctx, f"expect config data to be dict but got {ranks}") + return make_reply(ReturnCode.BAD_TASK_DATA) + + me = fl_ctx.get_identity_name() + my_rank = ranks.get(me) + if my_rank is None: + self.log_error(fl_ctx, f"missing rank for me ({me}) in config data") + return make_reply(ReturnCode.BAD_TASK_DATA) + + self.log_info(fl_ctx, f"got my rank: {my_rank}") + + num_rounds = shareable.get(Constant.CONF_KEY_NUM_ROUNDS) + if not num_rounds: + self.log_error(fl_ctx, f"missing {Constant.CONF_KEY_NUM_ROUNDS} from config") + return make_reply(ReturnCode.BAD_TASK_DATA) + + world_size = len(ranks) + + # configure the XGB client target via the adaptor + self.adaptor.configure( + { + Constant.CONF_KEY_RANK: my_rank, + Constant.CONF_KEY_NUM_ROUNDS: num_rounds, + Constant.CONF_KEY_WORLD_SIZE: world_size, + }, + fl_ctx, + ) + return make_reply(ReturnCode.OK) + elif task_name == self.start_task_name: + # start adaptor + try: + self.adaptor.start(fl_ctx) + except Exception as ex: + self.log_exception(fl_ctx, f"failed to start adaptor: {secure_format_exception(ex)}") + return make_reply(ReturnCode.EXECUTION_EXCEPTION) + + # start to monitor the XGB target via the adaptor + self.adaptor.monitor_target(fl_ctx, self._notify_client_done) + return make_reply(ReturnCode.OK) + else: + self.log_error(fl_ctx, f"ignored unsupported {task_name}") + return make_reply(ReturnCode.TASK_UNSUPPORTED) + + def _notify_client_done(self, rc, fl_ctx: FLContext): + """This is called when the XGB client target is done. + We send a message to the FL server telling it that this client is done. + + Args: + rc: the return code from the XGB client target + fl_ctx: FL context + + Returns: None + + """ + if rc != 0: + self.log_error(fl_ctx, f"XGB Client stopped with RC {rc}") + else: + self.log_info(fl_ctx, "XGB Client Stopped") + + # tell server that this client is done + engine = fl_ctx.get_engine() + req = Shareable() + req[Constant.MSG_KEY_EXIT_CODE] = rc + engine.send_aux_request( + targets=[FQCN.ROOT_SERVER], + topic=Constant.TOPIC_CLIENT_DONE, + request=req, + timeout=0, # fire and forget + fl_ctx=fl_ctx, + optional=True, + ) diff --git a/nvflare/app_common/xgb/fed_controller.py b/nvflare/app_common/xgb/fed_controller.py new file mode 100644 index 0000000000..959fb6dcce --- /dev/null +++ b/nvflare/app_common/xgb/fed_controller.py @@ -0,0 +1,62 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from nvflare.apis.fl_context import FLContext +from nvflare.app_common.xgb.adaptors.grpc_server_adaptor import GrpcServerAdaptor +from nvflare.app_common.xgb.runners.xgb_server_runner import XGBServerRunner + +from .controller import XGBController +from .defs import Constant + + +class XGBFedController(XGBController): + def __init__( + self, + num_rounds: int, + configure_task_name=Constant.CONFIG_TASK_NAME, + configure_task_timeout=Constant.CONFIG_TASK_TIMEOUT, + start_task_name=Constant.START_TASK_NAME, + start_task_timeout=Constant.START_TASK_TIMEOUT, + job_status_check_interval: float = Constant.JOB_STATUS_CHECK_INTERVAL, + max_client_op_interval: float = Constant.MAX_CLIENT_OP_INTERVAL, + progress_timeout: float = Constant.WORKFLOW_PROGRESS_TIMEOUT, + client_ranks=None, + int_client_grpc_options=None, + in_process=True, + ): + XGBController.__init__( + self, + adaptor_component_id="", + num_rounds=num_rounds, + configure_task_name=configure_task_name, + configure_task_timeout=configure_task_timeout, + start_task_name=start_task_name, + start_task_timeout=start_task_timeout, + job_status_check_interval=job_status_check_interval, + max_client_op_interval=max_client_op_interval, + progress_timeout=progress_timeout, + client_ranks=client_ranks, + ) + self.int_client_grpc_options = int_client_grpc_options + self.in_process = in_process + + def get_adaptor(self, fl_ctx: FLContext): + runner = XGBServerRunner() + runner.initialize(fl_ctx) + adaptor = GrpcServerAdaptor( + int_client_grpc_options=self.int_client_grpc_options, + in_process=self.in_process, + ) + adaptor.set_runner(runner) + return adaptor diff --git a/nvflare/app_common/xgb/fed_executor.py b/nvflare/app_common/xgb/fed_executor.py new file mode 100644 index 0000000000..f64ecf5f32 --- /dev/null +++ b/nvflare/app_common/xgb/fed_executor.py @@ -0,0 +1,63 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from nvflare.apis.fl_context import FLContext +from nvflare.app_common.xgb.adaptors.grpc_client_adaptor import GrpcClientAdaptor +from nvflare.app_common.xgb.runners.xgb_client_runner import XGBClientRunner + +from .executor import XGBExecutor + + +class FedXGBHistogramExecutor(XGBExecutor): + def __init__( + self, + early_stopping_rounds, + xgb_params: dict, + data_loader_id: str, + verbose_eval=False, + use_gpus=False, + int_server_grpc_options=None, + req_timeout=10.0, + model_file_name="model.json", + in_process=True, + ): + XGBExecutor.__init__( + self, + adaptor_component_id="", + req_timeout=req_timeout, + ) + self.early_stopping_rounds = early_stopping_rounds + self.xgb_params = xgb_params + self.data_loader_id = data_loader_id + self.verbose_eval = verbose_eval + self.use_gpus = use_gpus + self.int_server_grpc_options = int_server_grpc_options + self.model_file_name = model_file_name + self.in_process = in_process + + def get_adaptor(self, fl_ctx: FLContext): + runner = XGBClientRunner( + data_loader_id=self.data_loader_id, + early_stopping_rounds=self.early_stopping_rounds, + xgb_params=self.xgb_params, + verbose_eval=self.verbose_eval, + use_gpus=self.use_gpus, + model_file_name=self.model_file_name, + ) + runner.initialize(fl_ctx) + adaptor = GrpcClientAdaptor( + int_server_grpc_options=self.int_server_grpc_options, + in_process=self.in_process, + ) + adaptor.set_runner(runner) + return adaptor diff --git a/nvflare/app_common/xgb/grpc_client.py b/nvflare/app_common/xgb/grpc_client.py new file mode 100644 index 0000000000..7b0135e9e6 --- /dev/null +++ b/nvflare/app_common/xgb/grpc_client.py @@ -0,0 +1,178 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import grpc + +import nvflare.app_common.xgb.proto.federated_pb2 as pb2 +from nvflare.app_common.xgb.defs import GRPC_DEFAULT_OPTIONS +from nvflare.app_common.xgb.proto.federated_pb2_grpc import FederatedStub +from nvflare.fuel.utils.obj_utils import get_logger + + +class GrpcClient: + """This class implements a gRPC XGB Client that is capable of sending XGB operations to a gRPC XGB Server.""" + + def __init__(self, server_addr, grpc_options=None): + """Constructor + + Args: + server_addr: address of the gRPC server to connect to + grpc_options: gRPC options for the gRPC client + """ + if not grpc_options: + grpc_options = GRPC_DEFAULT_OPTIONS + + self.stub = None + self.channel = None + self.server_addr = server_addr + self.grpc_options = grpc_options + self.started = False + self.logger = get_logger(self) + + def start(self, ready_timeout=10): + """Start the gRPC client and wait for the server to be ready. + + Args: + ready_timeout: how long to wait for the server to be ready + + Returns: None + + """ + if self.started: + return + + self.started = True + + # TBD: need to support secure channel as well + self.channel = grpc.insecure_channel(self.server_addr, options=self.grpc_options) + self.stub = FederatedStub(self.channel) + + # wait for channel ready + try: + grpc.channel_ready_future(self.channel).result(timeout=ready_timeout) + except grpc.FutureTimeoutError: + raise RuntimeError(f"cannot connect to server after {ready_timeout} seconds") + + def send_allgather(self, seq_num, rank, data: bytes): + """Send Allgather request to gRPC server + + Args: + seq_num: sequence number + rank: rank of the client + data: the send_buf data + + Returns: an AllgatherReply object; or None if processing error is encountered + + """ + req = pb2.AllgatherRequest( + sequence_number=seq_num, + rank=rank, + send_buffer=data, + ) + + result = self.stub.Allgather(req) + if not isinstance(result, pb2.AllgatherReply): + self.logger.error(f"expect reply to be pb2.AllgatherReply but got {type(result)}") + return None + return result + + def send_allgatherv(self, seq_num, rank, data: bytes): + """Send AllgatherV request to gRPC server + + Args: + seq_num: sequence number + rank: rank of the client + data: the send_buf data + + Returns: an AllgatherVReply object; or None if processing error is encountered + + """ + req = pb2.AllgatherVRequest( + sequence_number=seq_num, + rank=rank, + send_buffer=data, + ) + + result = self.stub.AllgatherV(req) + if not isinstance(result, pb2.AllgatherVReply): + self.logger.error(f"expect reply to be pb2.AllgatherVReply but got {type(result)}") + return None + return result + + def send_allreduce(self, seq_num, rank, data: bytes, data_type, reduce_op): + """Send Allreduce request to gRPC server + + Args: + seq_num: sequence number + rank: rank of the client + data: the send_buf data + data_type: data type of the input + reduce_op: reduce op to be performed + + Returns: an AllreduceReply object; or None if processing error is encountered + + """ + req = pb2.AllreduceRequest( + sequence_number=seq_num, + rank=rank, + send_buffer=data, + data_type=data_type, + reduce_operation=reduce_op, + ) + + result = self.stub.Allreduce(req) + if not isinstance(result, pb2.AllreduceReply): + self.logger.error(f"expect reply to be pb2.AllreduceReply but got {type(result)}") + return None + return result + + def send_broadcast(self, seq_num, rank, data: bytes, root): + """Send Broadcast request to gRPC server + + Args: + seq_num: sequence number + rank: rank of the client + data: the send_buf data + root: rank of the root + + Returns: a BroadcastReply object; or None if processing error is encountered + + """ + req = pb2.BroadcastRequest( + sequence_number=seq_num, + rank=rank, + send_buffer=data, + root=root, + ) + + result = self.stub.Broadcast(req) + if not isinstance(result, pb2.BroadcastReply): + self.logger.error(f"expect reply to be pb2.BroadcastReply but got {type(result)}") + return None + return result + + def stop(self): + """Stop the gRPC client + + Returns: None + + """ + ch = self.channel + self.channel = None # set to None in case another thread also tries to close. + if ch: + try: + ch.close() + except: + # ignore errors when closing the channel + pass diff --git a/nvflare/app_common/xgb/grpc_server.py b/nvflare/app_common/xgb/grpc_server.py new file mode 100644 index 0000000000..6ace57aecc --- /dev/null +++ b/nvflare/app_common/xgb/grpc_server.py @@ -0,0 +1,82 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import concurrent.futures as futures + +import grpc + +from nvflare.app_common.xgb.defs import GRPC_DEFAULT_OPTIONS +from nvflare.app_common.xgb.proto.federated_pb2_grpc import FederatedServicer, add_FederatedServicer_to_server +from nvflare.fuel.utils.obj_utils import get_logger +from nvflare.fuel.utils.validation_utils import check_object_type, check_positive_int +from nvflare.security.logging import secure_format_exception + + +class GrpcServer: + """This class implements a gRPC XGB Server that is capable of processing XGB operations.""" + + def __init__(self, addr, max_workers: int, grpc_options, servicer): + """Constructor + + Args: + addr: the listening address of the server + max_workers: max number of workers + grpc_options: gRPC options + servicer: the servicer that is capable of processing XGB requests + """ + if not grpc_options: + grpc_options = GRPC_DEFAULT_OPTIONS + + check_object_type("servicer", servicer, FederatedServicer) + check_positive_int("max_workers", max_workers) + self.grpc_server = grpc.server(futures.ThreadPoolExecutor(max_workers=max_workers), options=grpc_options) + add_FederatedServicer_to_server(servicer, self.grpc_server) + self.logger = get_logger(self) + + try: + # TBD: will be enhanced to support secure port + self.grpc_server.add_insecure_port(addr) + self.logger.info(f"XGBServer: added insecure port at {addr}") + except Exception as ex: + self.logger.error(f"cannot listen on {addr}: {secure_format_exception(ex)}") + + def start(self, no_blocking=False): + """Called to start the server + + Args: + no_blocking: whether blocking the current thread and wait for server termination + + Returns: None + + """ + self.logger.info("starting gRPC Server") + self.grpc_server.start() + if no_blocking: + # don't wait for server termination + return + else: + self.grpc_server.wait_for_termination() + self.logger.info("gRPC XGB server terminated") + + def shutdown(self): + """Shut down the gRPC server gracefully. + + Returns: + + """ + self.logger.info("shutting down gRPC XGB server") + server = self.grpc_server + self.grpc_server = None # in case another thread calls shutdown at the same time + if server: + server.stop(grace=0.5) diff --git a/nvflare/app_common/xgb/mock/__init__.py b/nvflare/app_common/xgb/mock/__init__.py new file mode 100644 index 0000000000..4fc50543f1 --- /dev/null +++ b/nvflare/app_common/xgb/mock/__init__.py @@ -0,0 +1,13 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/nvflare/app_common/xgb/mock/aggr_servicer.py b/nvflare/app_common/xgb/mock/aggr_servicer.py new file mode 100644 index 0000000000..142fa36a10 --- /dev/null +++ b/nvflare/app_common/xgb/mock/aggr_servicer.py @@ -0,0 +1,122 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import threading + +import nvflare.app_common.xgb.proto.federated_pb2 as pb2 +from nvflare.app_common.xgb.proto.federated_pb2_grpc import FederatedServicer +from nvflare.fuel.utils.obj_utils import get_logger + + +class ReqWaiter: + def __init__(self, exp_num_clients: int, exp_seq: int, exp_op): + self.exp_num_clients = exp_num_clients + self.exp_seq = exp_seq + self.exp_op = exp_op + self.reqs = {} + self.result = {} + self.waiter = threading.Event() + + def add_request(self, op: str, rank, seq, req): + if seq != self.exp_seq: + raise RuntimeError(f"expecting seq {self.exp_seq} from {rank=} but got {seq}") + + if op != self.exp_op: + raise RuntimeError(f"expecting op {self.exp_op} from {rank=} but got {op}") + + if rank in self.reqs: + raise RuntimeError(f"duplicate request from {op=} {rank=} {seq=}") + + self.reqs[rank] = req + + if isinstance(req, pb2.AllgatherRequest): + reply = pb2.AllgatherReply(receive_buffer=req.send_buffer) + elif isinstance(req, pb2.AllgatherVRequest): + reply = pb2.AllgatherVReply(receive_buffer=req.send_buffer) + elif isinstance(req, pb2.AllreduceRequest): + reply = pb2.AllreduceReply(receive_buffer=req.send_buffer) + elif isinstance(req, pb2.BroadcastRequest): + reply = pb2.BroadcastReply(receive_buffer=req.send_buffer) + else: + raise RuntimeError(f"unknown request type {type(req)}") + self.result[rank] = reply + if len(self.reqs) == self.exp_num_clients: + self.waiter.set() + + def wait(self, timeout): + return self.waiter.wait(timeout) + + +class AggrServicer(FederatedServicer): + def __init__(self, num_clients, aggr_timeout=10.0): + self.logger = get_logger(self) + self.num_clients = num_clients + self.aggr_timeout = aggr_timeout + self.req_lock = threading.Lock() + self.req_waiter = None + + def _wait_for_result(self, op, rank, seq, request): + with self.req_lock: + if not self.req_waiter: + self.logger.info(f"setting new waiter: {seq=} {op=}") + self.req_waiter = ReqWaiter( + exp_num_clients=self.num_clients, + exp_seq=seq, + exp_op=op, + ) + self.req_waiter.add_request(op, rank, seq, request) + if not self.req_waiter.wait(self.aggr_timeout): + self.logger.error(f"results not received from all ranks after {self.aggr_timeout} seconds") + self.logger.info(f"for {rank=}: results remaining: {self.req_waiter.result.keys()}") + with self.req_lock: + result = self.req_waiter.result.pop(rank, None) + if len(self.req_waiter.result) == 0: + self.logger.info("all results are retrieved - reset req_waiter to None") + self.req_waiter = None + return result + + def Allgather(self, request: pb2.AllgatherRequest, context): + seq = request.sequence_number + rank = request.rank + data = request.send_buffer + op = "Allgather" + self.logger.info(f"got {op}: {seq=} {rank=} data_size={len(data)}") + return self._wait_for_result(op, rank, seq, request) + + def AllgatherV(self, request: pb2.AllgatherVRequest, context): + seq = request.sequence_number + rank = request.rank + data = request.send_buffer + op = "AllgatherV" + self.logger.info(f"got {op}: {seq=} {rank=} data_size={len(data)}") + return self._wait_for_result(op, rank, seq, request) + + def Allreduce(self, request: pb2.AllreduceRequest, context): + seq = request.sequence_number + rank = request.rank + data = request.send_buffer + reduce_op = request.reduce_operation + data_type = request.data_type + op = "Allreduce" + self.logger.info(f"got {op}: {seq=} {rank=} {reduce_op=} {data_type=} data_size={len(data)}") + return self._wait_for_result(op, rank, seq, request) + + def Broadcast(self, request: pb2.BroadcastRequest, context): + seq = request.sequence_number + rank = request.rank + data = request.send_buffer + root = request.root + op = "Broadcast" + self.logger.info(f"got {op}: {seq=} {rank=} {root=} data_size={len(data)}") + return self._wait_for_result(op, rank, seq, request) diff --git a/nvflare/app_common/xgb/mock/mock_client_runner.py b/nvflare/app_common/xgb/mock/mock_client_runner.py new file mode 100644 index 0000000000..7e16dd6f17 --- /dev/null +++ b/nvflare/app_common/xgb/mock/mock_client_runner.py @@ -0,0 +1,123 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import os +import time + +import nvflare.app_common.xgb.proto.federated_pb2 as pb2 +from nvflare.apis.fl_component import FLComponent +from nvflare.app_common.xgb.defs import Constant +from nvflare.app_common.xgb.grpc_client import GrpcClient +from nvflare.app_common.xgb.runners.xgb_runner import XGBRunner + + +class MockClientRunner(XGBRunner, FLComponent): + def __init__(self): + FLComponent.__init__(self) + self.training_stopped = False + self.asked_to_stop = False + + def run(self, ctx: dict): + server_addr = ctx.get(Constant.RUNNER_CTX_SERVER_ADDR) + rank = ctx.get(Constant.RUNNER_CTX_RANK) + num_rounds = ctx.get(Constant.RUNNER_CTX_NUM_ROUNDS) + + client = GrpcClient(server_addr=server_addr) + client.start() + + rank = rank + seq = 0 + total_time = 0 + total_reqs = 0 + for i in range(num_rounds): + if self.asked_to_stop: + self.logger.info("training aborted") + self.training_stopped = True + return + + self.logger.info(f"Test round {i}") + data = os.urandom(1000000) + + self.logger.info("sending allgather") + start = time.time() + result = client.send_allgather(seq_num=seq + 1, rank=rank, data=data) + total_reqs += 1 + total_time += time.time() - start + if not isinstance(result, pb2.AllgatherReply): + self.logger.error(f"expect reply to be pb2.AllgatherReply but got {type(result)}") + elif result.receive_buffer != data: + self.logger.error("allgather result does not match request") + else: + self.logger.info("OK: allgather result matches request!") + + self.logger.info("sending allgatherV") + start = time.time() + result = client.send_allgatherv(seq_num=seq + 2, rank=rank, data=data) + total_reqs += 1 + total_time += time.time() - start + if not isinstance(result, pb2.AllgatherVReply): + self.logger.error(f"expect reply to be pb2.AllgatherVReply but got {type(result)}") + elif result.receive_buffer != data: + self.logger.error("allgatherV result does not match request") + else: + self.logger.info("OK: allgatherV result matches request!") + + self.logger.info("sending allreduce") + start = time.time() + result = client.send_allreduce( + seq_num=seq + 3, + rank=rank, + data=data, + reduce_op=2, + data_type=2, + ) + total_reqs += 1 + total_time += time.time() - start + if not isinstance(result, pb2.AllreduceReply): + self.logger.error(f"expect reply to be pb2.AllreduceReply but got {type(result)}") + elif result.receive_buffer != data: + self.logger.error("allreduce result does not match request") + else: + self.logger.info("OK: allreduce result matches request!") + print("OK: allreduce result matches request!") + + self.logger.info("sending broadcast") + start = time.time() + result = client.send_broadcast( + seq_num=seq + 4, + rank=rank, + data=data, + root=3, + ) + total_reqs += 1 + total_time += time.time() - start + if not isinstance(result, pb2.BroadcastReply): + self.logger.error(f"expect reply to be pb2.BroadcastReply but got {type(result)}") + elif result.receive_buffer != data: + self.logger.error("ERROR: broadcast result does not match request") + else: + self.logger.info("OK: broadcast result matches request!") + + seq += 4 + time.sleep(1.0) + + time_per_req = total_time / total_reqs + self.logger.info(f"DONE: {total_reqs=} {total_time=} {time_per_req=}") + print(f"DONE: {total_reqs=} {total_time=} {time_per_req=}") + self.training_stopped = True + + def stop(self): + self.asked_to_stop = True + + def is_stopped(self) -> (bool, int): + return self.training_stopped, 0 diff --git a/nvflare/app_common/xgb/mock/mock_controller.py b/nvflare/app_common/xgb/mock/mock_controller.py new file mode 100644 index 0000000000..904ba66f83 --- /dev/null +++ b/nvflare/app_common/xgb/mock/mock_controller.py @@ -0,0 +1,61 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from nvflare.apis.fl_context import FLContext +from nvflare.app_common.xgb.adaptors.grpc_server_adaptor import GrpcServerAdaptor +from nvflare.app_common.xgb.controller import XGBController +from nvflare.app_common.xgb.defs import Constant +from nvflare.app_common.xgb.mock.mock_server_runner import MockServerRunner + + +class MockXGBController(XGBController): + def __init__( + self, + num_rounds: int, + configure_task_name=Constant.CONFIG_TASK_NAME, + configure_task_timeout=Constant.CONFIG_TASK_TIMEOUT, + start_task_name=Constant.START_TASK_NAME, + start_task_timeout=Constant.START_TASK_TIMEOUT, + job_status_check_interval: float = Constant.JOB_STATUS_CHECK_INTERVAL, + max_client_op_interval: float = Constant.MAX_CLIENT_OP_INTERVAL, + progress_timeout: float = Constant.WORKFLOW_PROGRESS_TIMEOUT, + client_ranks=None, + int_client_grpc_options=None, + in_process=True, + ): + XGBController.__init__( + self, + adaptor_component_id="", + num_rounds=num_rounds, + configure_task_name=configure_task_name, + configure_task_timeout=configure_task_timeout, + start_task_name=start_task_name, + start_task_timeout=start_task_timeout, + job_status_check_interval=job_status_check_interval, + max_client_op_interval=max_client_op_interval, + progress_timeout=progress_timeout, + client_ranks=client_ranks, + ) + self.int_client_grpc_options = int_client_grpc_options + self.in_process = in_process + + def get_adaptor(self, fl_ctx: FLContext): + runner = MockServerRunner() + runner.initialize(fl_ctx) + adaptor = GrpcServerAdaptor( + int_client_grpc_options=self.int_client_grpc_options, + in_process=self.in_process, + ) + adaptor.set_runner(runner) + return adaptor diff --git a/nvflare/app_common/xgb/mock/mock_executor.py b/nvflare/app_common/xgb/mock/mock_executor.py new file mode 100644 index 0000000000..d384ea2c63 --- /dev/null +++ b/nvflare/app_common/xgb/mock/mock_executor.py @@ -0,0 +1,43 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from nvflare.apis.fl_context import FLContext +from nvflare.app_common.xgb.adaptors.grpc_client_adaptor import GrpcClientAdaptor +from nvflare.app_common.xgb.executor import XGBExecutor +from nvflare.app_common.xgb.mock.mock_client_runner import MockClientRunner + + +class MockXGBExecutor(XGBExecutor): + def __init__( + self, + int_server_grpc_options=None, + req_timeout=10.0, + in_process=True, + ): + XGBExecutor.__init__( + self, + adaptor_component_id="", + req_timeout=req_timeout, + ) + self.int_server_grpc_options = int_server_grpc_options + self.in_process = in_process + + def get_adaptor(self, fl_ctx: FLContext): + runner = MockClientRunner() + runner.initialize(fl_ctx) + adaptor = GrpcClientAdaptor( + int_server_grpc_options=self.int_server_grpc_options, + in_process=self.in_process, + ) + adaptor.set_runner(runner) + return adaptor diff --git a/nvflare/app_common/xgb/mock/mock_server_runner.py b/nvflare/app_common/xgb/mock/mock_server_runner.py new file mode 100644 index 0000000000..8539b3c290 --- /dev/null +++ b/nvflare/app_common/xgb/mock/mock_server_runner.py @@ -0,0 +1,46 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from nvflare.app_common.xgb.defs import Constant +from nvflare.app_common.xgb.grpc_server import GrpcServer +from nvflare.app_common.xgb.mock.aggr_servicer import AggrServicer +from nvflare.app_common.xgb.runners.xgb_runner import XGBRunner + + +class MockServerRunner(XGBRunner): + def __init__(self, server_max_workers=10): + self.server_max_workers = server_max_workers + self._stopped = False + self._server = None + + def run(self, ctx: dict): + world_size = ctx.get(Constant.RUNNER_CTX_WORLD_SIZE) + addr = ctx.get(Constant.RUNNER_CTX_SERVER_ADDR) + + self._server = GrpcServer( + addr, + max_workers=self.server_max_workers, + grpc_options=None, + servicer=AggrServicer(num_clients=world_size), + ) + self._server.start(no_blocking=False) + + def stop(self): + s = self._server + self._server = None + if s: + s.shutdown() + self._stopped = True + + def is_stopped(self) -> (bool, int): + return self._stopped, 0 diff --git a/nvflare/app_common/xgb/mock/run_client.py b/nvflare/app_common/xgb/mock/run_client.py new file mode 100644 index 0000000000..bbd6551322 --- /dev/null +++ b/nvflare/app_common/xgb/mock/run_client.py @@ -0,0 +1,108 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import os +import time + +import nvflare.app_common.xgb.proto.federated_pb2 as pb2 +from nvflare.app_common.xgb.grpc_client import GrpcClient + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--addr", "-a", type=str, help="server address", required=True) + parser.add_argument("--rank", "-r", type=int, help="client rank", required=True) + parser.add_argument("--num_rounds", "-n", type=int, help="number of rounds", required=True) + + args = parser.parse_args() + client = GrpcClient(server_addr=args.addr) + client.start() + + rank = args.rank + seq = 0 + total_time = 0 + total_reqs = 0 + for i in range(args.num_rounds): + print(f"Test round {i}") + data = os.urandom(1000000) + + print("sending allgather") + start = time.time() + result = client.send_allgather(seq_num=seq + 1, rank=rank, data=data) + total_reqs += 1 + total_time += time.time() - start + if not isinstance(result, pb2.AllgatherReply): + print(f"expect reply to be pb2.AllgatherReply but got {type(result)}") + elif result.receive_buffer != data: + print("ERROR: allgather result does not match request") + else: + print("OK: allgather result matches request!") + + print("sending allgatherV") + start = time.time() + result = client.send_allgatherv(seq_num=seq + 2, rank=rank, data=data) + total_reqs += 1 + total_time += time.time() - start + if not isinstance(result, pb2.AllgatherVReply): + print(f"expect reply to be pb2.AllgatherVReply but got {type(result)}") + elif result.receive_buffer != data: + print("ERROR: allgatherV result does not match request") + else: + print("OK: allgatherV result matches request!") + + print("sending allreduce") + start = time.time() + result = client.send_allreduce( + seq_num=seq + 3, + rank=rank, + data=data, + reduce_op=2, + data_type=2, + ) + total_reqs += 1 + total_time += time.time() - start + if not isinstance(result, pb2.AllreduceReply): + print(f"expect reply to be pb2.AllreduceReply but got {type(result)}") + elif result.receive_buffer != data: + print("ERROR: allreduce result does not match request") + else: + print("OK: allreduce result matches request!") + + print("sending broadcast") + start = time.time() + result = client.send_broadcast( + seq_num=seq + 4, + rank=rank, + data=data, + root=3, + ) + total_reqs += 1 + total_time += time.time() - start + if not isinstance(result, pb2.BroadcastReply): + print(f"expect reply to be pb2.BroadcastReply but got {type(result)}") + elif result.receive_buffer != data: + print("ERROR: broadcast result does not match request") + else: + print("OK: broadcast result matches request!") + + seq += 4 + time.sleep(1.0) + + time_per_req = total_time / total_reqs + print(f"DONE: {total_reqs=} {total_time=} {time_per_req=}") + + +if __name__ == "__main__": + main() diff --git a/nvflare/app_common/xgb/mock/run_server.py b/nvflare/app_common/xgb/mock/run_server.py new file mode 100644 index 0000000000..6ac4ae02e3 --- /dev/null +++ b/nvflare/app_common/xgb/mock/run_server.py @@ -0,0 +1,43 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import logging + +from nvflare.app_common.xgb.grpc_server import GrpcServer +from nvflare.app_common.xgb.mock.aggr_servicer import AggrServicer + + +def main(): + logging.basicConfig() + logging.getLogger().setLevel(logging.INFO) + + parser = argparse.ArgumentParser() + parser.add_argument("--addr", "-a", type=str, help="server address", required=True) + parser.add_argument("--num_clients", "-c", type=int, help="number of clients", required=True) + parser.add_argument("--max_workers", "-w", type=int, help="max number of workers", required=False, default=20) + + args = parser.parse_args() + print(f"starting server at {args.addr} max_workers={args.max_workers}") + server = GrpcServer( + args.addr, + max_workers=args.max_workers, + grpc_options=None, + servicer=AggrServicer(num_clients=args.num_clients), + ) + server.start() + + +if __name__ == "__main__": + main() diff --git a/nvflare/app_common/xgb/proto/__init__.py b/nvflare/app_common/xgb/proto/__init__.py new file mode 100644 index 0000000000..4fc50543f1 --- /dev/null +++ b/nvflare/app_common/xgb/proto/__init__.py @@ -0,0 +1,13 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/nvflare/app_common/xgb/proto/federated.proto b/nvflare/app_common/xgb/proto/federated.proto new file mode 100644 index 0000000000..f412204813 --- /dev/null +++ b/nvflare/app_common/xgb/proto/federated.proto @@ -0,0 +1,85 @@ +/*! + * Copyright 2022-2023 XGBoost contributors + */ +syntax = "proto3"; + +package xgboost.collective.federated; + +service Federated { + rpc Allgather(AllgatherRequest) returns (AllgatherReply) {} + rpc AllgatherV(AllgatherVRequest) returns (AllgatherVReply) {} + rpc Allreduce(AllreduceRequest) returns (AllreduceReply) {} + rpc Broadcast(BroadcastRequest) returns (BroadcastReply) {} +} + +enum DataType { + HALF = 0; + FLOAT = 1; + DOUBLE = 2; + LONG_DOUBLE = 3; + INT8 = 4; + INT16 = 5; + INT32 = 6; + INT64 = 7; + UINT8 = 8; + UINT16 = 9; + UINT32 = 10; + UINT64 = 11; +} + +enum ReduceOperation { + MAX = 0; + MIN = 1; + SUM = 2; + BITWISE_AND = 3; + BITWISE_OR = 4; + BITWISE_XOR = 5; +} + +message AllgatherRequest { + // An incrementing counter that is unique to each round to operations. + uint64 sequence_number = 1; + int32 rank = 2; + bytes send_buffer = 3; +} + +message AllgatherReply { + bytes receive_buffer = 1; +} + +message AllgatherVRequest { + // An incrementing counter that is unique to each round to operations. + uint64 sequence_number = 1; + int32 rank = 2; + bytes send_buffer = 3; +} + +message AllgatherVReply { + bytes receive_buffer = 1; +} + +message AllreduceRequest { + // An incrementing counter that is unique to each round to operations. + uint64 sequence_number = 1; + int32 rank = 2; + bytes send_buffer = 3; + DataType data_type = 4; + ReduceOperation reduce_operation = 5; +} + +message AllreduceReply { + bytes receive_buffer = 1; +} + +message BroadcastRequest { + // An incrementing counter that is unique to each round to operations. + uint64 sequence_number = 1; + int32 rank = 2; + bytes send_buffer = 3; + // The root rank to broadcast from. + int32 root = 4; +} + +message BroadcastReply { + bytes receive_buffer = 1; +} \ No newline at end of file diff --git a/nvflare/app_common/xgb/proto/federated_pb2.py b/nvflare/app_common/xgb/proto/federated_pb2.py new file mode 100644 index 0000000000..ba80c1e5d6 --- /dev/null +++ b/nvflare/app_common/xgb/proto/federated_pb2.py @@ -0,0 +1,59 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# -*- coding: utf-8 -*- +# Generated by the protocol buffer compiler. DO NOT EDIT! +# source: federated.proto +"""Generated protocol buffer code.""" +from google.protobuf.internal import builder as _builder +from google.protobuf import descriptor as _descriptor +from google.protobuf import descriptor_pool as _descriptor_pool +from google.protobuf import symbol_database as _symbol_database +# @@protoc_insertion_point(imports) + +_sym_db = _symbol_database.Default() + + + + +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x0f\x66\x65\x64\x65rated.proto\x12\x1cxgboost.collective.federated\"N\n\x10\x41llgatherRequest\x12\x17\n\x0fsequence_number\x18\x01 \x01(\x04\x12\x0c\n\x04rank\x18\x02 \x01(\x05\x12\x13\n\x0bsend_buffer\x18\x03 \x01(\x0c\"(\n\x0e\x41llgatherReply\x12\x16\n\x0ereceive_buffer\x18\x01 \x01(\x0c\"O\n\x11\x41llgatherVRequest\x12\x17\n\x0fsequence_number\x18\x01 \x01(\x04\x12\x0c\n\x04rank\x18\x02 \x01(\x05\x12\x13\n\x0bsend_buffer\x18\x03 \x01(\x0c\")\n\x0f\x41llgatherVReply\x12\x16\n\x0ereceive_buffer\x18\x01 \x01(\x0c\"\xd2\x01\n\x10\x41llreduceRequest\x12\x17\n\x0fsequence_number\x18\x01 \x01(\x04\x12\x0c\n\x04rank\x18\x02 \x01(\x05\x12\x13\n\x0bsend_buffer\x18\x03 \x01(\x0c\x12\x39\n\tdata_type\x18\x04 \x01(\x0e\x32&.xgboost.collective.federated.DataType\x12G\n\x10reduce_operation\x18\x05 \x01(\x0e\x32-.xgboost.collective.federated.ReduceOperation\"(\n\x0e\x41llreduceReply\x12\x16\n\x0ereceive_buffer\x18\x01 \x01(\x0c\"\\\n\x10\x42roadcastRequest\x12\x17\n\x0fsequence_number\x18\x01 \x01(\x04\x12\x0c\n\x04rank\x18\x02 \x01(\x05\x12\x13\n\x0bsend_buffer\x18\x03 \x01(\x0c\x12\x0c\n\x04root\x18\x04 \x01(\x05\"(\n\x0e\x42roadcastReply\x12\x16\n\x0ereceive_buffer\x18\x01 \x01(\x0c*\x96\x01\n\x08\x44\x61taType\x12\x08\n\x04HALF\x10\x00\x12\t\n\x05\x46LOAT\x10\x01\x12\n\n\x06\x44OUBLE\x10\x02\x12\x0f\n\x0bLONG_DOUBLE\x10\x03\x12\x08\n\x04INT8\x10\x04\x12\t\n\x05INT16\x10\x05\x12\t\n\x05INT32\x10\x06\x12\t\n\x05INT64\x10\x07\x12\t\n\x05UINT8\x10\x08\x12\n\n\x06UINT16\x10\t\x12\n\n\x06UINT32\x10\n\x12\n\n\x06UINT64\x10\x0b*^\n\x0fReduceOperation\x12\x07\n\x03MAX\x10\x00\x12\x07\n\x03MIN\x10\x01\x12\x07\n\x03SUM\x10\x02\x12\x0f\n\x0b\x42ITWISE_AND\x10\x03\x12\x0e\n\nBITWISE_OR\x10\x04\x12\x0f\n\x0b\x42ITWISE_XOR\x10\x05\x32\xc2\x03\n\tFederated\x12k\n\tAllgather\x12..xgboost.collective.federated.AllgatherRequest\x1a,.xgboost.collective.federated.AllgatherReply\"\x00\x12n\n\nAllgatherV\x12/.xgboost.collective.federated.AllgatherVRequest\x1a-.xgboost.collective.federated.AllgatherVReply\"\x00\x12k\n\tAllreduce\x12..xgboost.collective.federated.AllreduceRequest\x1a,.xgboost.collective.federated.AllreduceReply\"\x00\x12k\n\tBroadcast\x12..xgboost.collective.federated.BroadcastRequest\x1a,.xgboost.collective.federated.BroadcastReply\"\x00\x62\x06proto3') + +_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, globals()) +_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'federated_pb2', globals()) +if _descriptor._USE_C_DESCRIPTORS == False: + + DESCRIPTOR._options = None + _DATATYPE._serialized_start=687 + _DATATYPE._serialized_end=837 + _REDUCEOPERATION._serialized_start=839 + _REDUCEOPERATION._serialized_end=933 + _ALLGATHERREQUEST._serialized_start=49 + _ALLGATHERREQUEST._serialized_end=127 + _ALLGATHERREPLY._serialized_start=129 + _ALLGATHERREPLY._serialized_end=169 + _ALLGATHERVREQUEST._serialized_start=171 + _ALLGATHERVREQUEST._serialized_end=250 + _ALLGATHERVREPLY._serialized_start=252 + _ALLGATHERVREPLY._serialized_end=293 + _ALLREDUCEREQUEST._serialized_start=296 + _ALLREDUCEREQUEST._serialized_end=506 + _ALLREDUCEREPLY._serialized_start=508 + _ALLREDUCEREPLY._serialized_end=548 + _BROADCASTREQUEST._serialized_start=550 + _BROADCASTREQUEST._serialized_end=642 + _BROADCASTREPLY._serialized_start=644 + _BROADCASTREPLY._serialized_end=684 + _FEDERATED._serialized_start=936 + _FEDERATED._serialized_end=1386 +# @@protoc_insertion_point(module_scope) diff --git a/nvflare/app_common/xgb/proto/federated_pb2.pyi b/nvflare/app_common/xgb/proto/federated_pb2.pyi new file mode 100644 index 0000000000..8e2a7e740e --- /dev/null +++ b/nvflare/app_common/xgb/proto/federated_pb2.pyi @@ -0,0 +1,100 @@ +from google.protobuf.internal import enum_type_wrapper as _enum_type_wrapper +from google.protobuf import descriptor as _descriptor +from google.protobuf import message as _message +from typing import ClassVar as _ClassVar, Optional as _Optional, Union as _Union + +BITWISE_AND: ReduceOperation +BITWISE_OR: ReduceOperation +BITWISE_XOR: ReduceOperation +DESCRIPTOR: _descriptor.FileDescriptor +DOUBLE: DataType +FLOAT: DataType +HALF: DataType +INT16: DataType +INT32: DataType +INT64: DataType +INT8: DataType +LONG_DOUBLE: DataType +MAX: ReduceOperation +MIN: ReduceOperation +SUM: ReduceOperation +UINT16: DataType +UINT32: DataType +UINT64: DataType +UINT8: DataType + +class AllgatherReply(_message.Message): + __slots__ = ["receive_buffer"] + RECEIVE_BUFFER_FIELD_NUMBER: _ClassVar[int] + receive_buffer: bytes + def __init__(self, receive_buffer: _Optional[bytes] = ...) -> None: ... + +class AllgatherRequest(_message.Message): + __slots__ = ["rank", "send_buffer", "sequence_number"] + RANK_FIELD_NUMBER: _ClassVar[int] + SEND_BUFFER_FIELD_NUMBER: _ClassVar[int] + SEQUENCE_NUMBER_FIELD_NUMBER: _ClassVar[int] + rank: int + send_buffer: bytes + sequence_number: int + def __init__(self, sequence_number: _Optional[int] = ..., rank: _Optional[int] = ..., send_buffer: _Optional[bytes] = ...) -> None: ... + +class AllgatherVReply(_message.Message): + __slots__ = ["receive_buffer"] + RECEIVE_BUFFER_FIELD_NUMBER: _ClassVar[int] + receive_buffer: bytes + def __init__(self, receive_buffer: _Optional[bytes] = ...) -> None: ... + +class AllgatherVRequest(_message.Message): + __slots__ = ["rank", "send_buffer", "sequence_number"] + RANK_FIELD_NUMBER: _ClassVar[int] + SEND_BUFFER_FIELD_NUMBER: _ClassVar[int] + SEQUENCE_NUMBER_FIELD_NUMBER: _ClassVar[int] + rank: int + send_buffer: bytes + sequence_number: int + def __init__(self, sequence_number: _Optional[int] = ..., rank: _Optional[int] = ..., send_buffer: _Optional[bytes] = ...) -> None: ... + +class AllreduceReply(_message.Message): + __slots__ = ["receive_buffer"] + RECEIVE_BUFFER_FIELD_NUMBER: _ClassVar[int] + receive_buffer: bytes + def __init__(self, receive_buffer: _Optional[bytes] = ...) -> None: ... + +class AllreduceRequest(_message.Message): + __slots__ = ["data_type", "rank", "reduce_operation", "send_buffer", "sequence_number"] + DATA_TYPE_FIELD_NUMBER: _ClassVar[int] + RANK_FIELD_NUMBER: _ClassVar[int] + REDUCE_OPERATION_FIELD_NUMBER: _ClassVar[int] + SEND_BUFFER_FIELD_NUMBER: _ClassVar[int] + SEQUENCE_NUMBER_FIELD_NUMBER: _ClassVar[int] + data_type: DataType + rank: int + reduce_operation: ReduceOperation + send_buffer: bytes + sequence_number: int + def __init__(self, sequence_number: _Optional[int] = ..., rank: _Optional[int] = ..., send_buffer: _Optional[bytes] = ..., data_type: _Optional[_Union[DataType, str]] = ..., reduce_operation: _Optional[_Union[ReduceOperation, str]] = ...) -> None: ... + +class BroadcastReply(_message.Message): + __slots__ = ["receive_buffer"] + RECEIVE_BUFFER_FIELD_NUMBER: _ClassVar[int] + receive_buffer: bytes + def __init__(self, receive_buffer: _Optional[bytes] = ...) -> None: ... + +class BroadcastRequest(_message.Message): + __slots__ = ["rank", "root", "send_buffer", "sequence_number"] + RANK_FIELD_NUMBER: _ClassVar[int] + ROOT_FIELD_NUMBER: _ClassVar[int] + SEND_BUFFER_FIELD_NUMBER: _ClassVar[int] + SEQUENCE_NUMBER_FIELD_NUMBER: _ClassVar[int] + rank: int + root: int + send_buffer: bytes + sequence_number: int + def __init__(self, sequence_number: _Optional[int] = ..., rank: _Optional[int] = ..., send_buffer: _Optional[bytes] = ..., root: _Optional[int] = ...) -> None: ... + +class DataType(int, metaclass=_enum_type_wrapper.EnumTypeWrapper): + __slots__ = [] + +class ReduceOperation(int, metaclass=_enum_type_wrapper.EnumTypeWrapper): + __slots__ = [] diff --git a/nvflare/app_common/xgb/proto/federated_pb2_grpc.py b/nvflare/app_common/xgb/proto/federated_pb2_grpc.py new file mode 100644 index 0000000000..206d8474da --- /dev/null +++ b/nvflare/app_common/xgb/proto/federated_pb2_grpc.py @@ -0,0 +1,179 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! +"""Client and server classes corresponding to protobuf-defined services.""" +import grpc + +import nvflare.app_common.xgb.proto.federated_pb2 as federated__pb2 + + +class FederatedStub(object): + """Missing associated documentation comment in .proto file.""" + + def __init__(self, channel): + """Constructor. + + Args: + channel: A grpc.Channel. + """ + self.Allgather = channel.unary_unary( + '/xgboost.collective.federated.Federated/Allgather', + request_serializer=federated__pb2.AllgatherRequest.SerializeToString, + response_deserializer=federated__pb2.AllgatherReply.FromString, + ) + self.AllgatherV = channel.unary_unary( + '/xgboost.collective.federated.Federated/AllgatherV', + request_serializer=federated__pb2.AllgatherVRequest.SerializeToString, + response_deserializer=federated__pb2.AllgatherVReply.FromString, + ) + self.Allreduce = channel.unary_unary( + '/xgboost.collective.federated.Federated/Allreduce', + request_serializer=federated__pb2.AllreduceRequest.SerializeToString, + response_deserializer=federated__pb2.AllreduceReply.FromString, + ) + self.Broadcast = channel.unary_unary( + '/xgboost.collective.federated.Federated/Broadcast', + request_serializer=federated__pb2.BroadcastRequest.SerializeToString, + response_deserializer=federated__pb2.BroadcastReply.FromString, + ) + + +class FederatedServicer(object): + """Missing associated documentation comment in .proto file.""" + + def Allgather(self, request, context): + """Missing associated documentation comment in .proto file.""" + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def AllgatherV(self, request, context): + """Missing associated documentation comment in .proto file.""" + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def Allreduce(self, request, context): + """Missing associated documentation comment in .proto file.""" + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + def Broadcast(self, request, context): + """Missing associated documentation comment in .proto file.""" + context.set_code(grpc.StatusCode.UNIMPLEMENTED) + context.set_details('Method not implemented!') + raise NotImplementedError('Method not implemented!') + + +def add_FederatedServicer_to_server(servicer, server): + rpc_method_handlers = { + 'Allgather': grpc.unary_unary_rpc_method_handler( + servicer.Allgather, + request_deserializer=federated__pb2.AllgatherRequest.FromString, + response_serializer=federated__pb2.AllgatherReply.SerializeToString, + ), + 'AllgatherV': grpc.unary_unary_rpc_method_handler( + servicer.AllgatherV, + request_deserializer=federated__pb2.AllgatherVRequest.FromString, + response_serializer=federated__pb2.AllgatherVReply.SerializeToString, + ), + 'Allreduce': grpc.unary_unary_rpc_method_handler( + servicer.Allreduce, + request_deserializer=federated__pb2.AllreduceRequest.FromString, + response_serializer=federated__pb2.AllreduceReply.SerializeToString, + ), + 'Broadcast': grpc.unary_unary_rpc_method_handler( + servicer.Broadcast, + request_deserializer=federated__pb2.BroadcastRequest.FromString, + response_serializer=federated__pb2.BroadcastReply.SerializeToString, + ), + } + generic_handler = grpc.method_handlers_generic_handler( + 'xgboost.collective.federated.Federated', rpc_method_handlers) + server.add_generic_rpc_handlers((generic_handler,)) + + + # This class is part of an EXPERIMENTAL API. +class Federated(object): + """Missing associated documentation comment in .proto file.""" + + @staticmethod + def Allgather(request, + target, + options=(), + channel_credentials=None, + call_credentials=None, + insecure=False, + compression=None, + wait_for_ready=None, + timeout=None, + metadata=None): + return grpc.experimental.unary_unary(request, target, '/xgboost.collective.federated.Federated/Allgather', + federated__pb2.AllgatherRequest.SerializeToString, + federated__pb2.AllgatherReply.FromString, + options, channel_credentials, + insecure, call_credentials, compression, wait_for_ready, timeout, metadata) + + @staticmethod + def AllgatherV(request, + target, + options=(), + channel_credentials=None, + call_credentials=None, + insecure=False, + compression=None, + wait_for_ready=None, + timeout=None, + metadata=None): + return grpc.experimental.unary_unary(request, target, '/xgboost.collective.federated.Federated/AllgatherV', + federated__pb2.AllgatherVRequest.SerializeToString, + federated__pb2.AllgatherVReply.FromString, + options, channel_credentials, + insecure, call_credentials, compression, wait_for_ready, timeout, metadata) + + @staticmethod + def Allreduce(request, + target, + options=(), + channel_credentials=None, + call_credentials=None, + insecure=False, + compression=None, + wait_for_ready=None, + timeout=None, + metadata=None): + return grpc.experimental.unary_unary(request, target, '/xgboost.collective.federated.Federated/Allreduce', + federated__pb2.AllreduceRequest.SerializeToString, + federated__pb2.AllreduceReply.FromString, + options, channel_credentials, + insecure, call_credentials, compression, wait_for_ready, timeout, metadata) + + @staticmethod + def Broadcast(request, + target, + options=(), + channel_credentials=None, + call_credentials=None, + insecure=False, + compression=None, + wait_for_ready=None, + timeout=None, + metadata=None): + return grpc.experimental.unary_unary(request, target, '/xgboost.collective.federated.Federated/Broadcast', + federated__pb2.BroadcastRequest.SerializeToString, + federated__pb2.BroadcastReply.FromString, + options, channel_credentials, + insecure, call_credentials, compression, wait_for_ready, timeout, metadata) diff --git a/nvflare/app_common/xgb/proto/gen_proto.sh b/nvflare/app_common/xgb/proto/gen_proto.sh new file mode 100644 index 0000000000..10afcf5b3b --- /dev/null +++ b/nvflare/app_common/xgb/proto/gen_proto.sh @@ -0,0 +1 @@ +python -m grpc_tools.protoc -I. --python_out=. --pyi_out=. --grpc_python_out=. federated.proto diff --git a/nvflare/app_common/xgb/runners/__init__.py b/nvflare/app_common/xgb/runners/__init__.py new file mode 100644 index 0000000000..4fc50543f1 --- /dev/null +++ b/nvflare/app_common/xgb/runners/__init__.py @@ -0,0 +1,13 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. diff --git a/nvflare/app_common/xgb/runners/xgb_client_runner.py b/nvflare/app_common/xgb/runners/xgb_client_runner.py new file mode 100644 index 0000000000..7771590813 --- /dev/null +++ b/nvflare/app_common/xgb/runners/xgb_client_runner.py @@ -0,0 +1,141 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import os + +import xgboost as xgb +from xgboost import callback + +from nvflare.apis.fl_component import FLComponent +from nvflare.apis.fl_context import FLContext +from nvflare.app_common.xgb.data_loader import XGBDataLoader +from nvflare.app_common.xgb.defs import Constant +from nvflare.app_common.xgb.runners.xgb_runner import XGBRunner +from nvflare.app_common.xgb.tb import TensorBoardCallback +from nvflare.app_common.xgb.xgb_params import XGBoostParams +from nvflare.fuel.utils.import_utils import optional_import +from nvflare.fuel.utils.obj_utils import get_logger + + +class XGBClientRunner(XGBRunner, FLComponent): + def __init__( + self, + data_loader_id: str, + early_stopping_rounds: int, + xgb_params: dict, + verbose_eval, + use_gpus, + model_file_name, + ): + FLComponent.__init__(self) + self.early_stopping_rounds = early_stopping_rounds + self.xgb_params = xgb_params + self.verbose_eval = verbose_eval + self.use_gpus = use_gpus + self.model_file_name = model_file_name + self.data_loader_id = data_loader_id + self.logger = get_logger(self) + + self._client_name = None + self._rank = None + self._world_size = None + self._num_rounds = None + self._server_addr = None + self._data_loader = None + self._tb_dir = None + self._model_dir = None + self._stopped = False + + def initialize(self, fl_ctx: FLContext): + engine = fl_ctx.get_engine() + self._data_loader = engine.get_component(self.data_loader_id) + if not isinstance(self._data_loader, XGBDataLoader): + self.system_panic(f"data_loader should be type XGBDataLoader but got {type(self._data_loader)}", fl_ctx) + + def _xgb_train(self, params: XGBoostParams, train_data, val_data) -> xgb.core.Booster: + """XGBoost training logic. + + Args: + params (XGBoostParams): xgboost parameters. + + Returns: + A xgboost booster. + """ + # Specify validations set to watch performance + watchlist = [(val_data, "eval"), (train_data, "train")] + + callbacks = [callback.EvaluationMonitor(rank=self._rank)] + tensorboard, flag = optional_import(module="torch.utils.tensorboard") + if flag and self._tb_dir: + callbacks.append(TensorBoardCallback(self._tb_dir, tensorboard)) + + # Run training, all the features in training API is available. + bst = xgb.train( + params.xgb_params, + train_data, + params.num_rounds, + evals=watchlist, + early_stopping_rounds=params.early_stopping_rounds, + verbose_eval=params.verbose_eval, + callbacks=callbacks, + ) + return bst + + def run(self, ctx: dict): + self._client_name = ctx.get(Constant.RUNNER_CTX_CLIENT_NAME) + self._rank = ctx.get(Constant.RUNNER_CTX_RANK) + self._world_size = ctx.get(Constant.RUNNER_CTX_WORLD_SIZE) + self._num_rounds = ctx.get(Constant.RUNNER_CTX_NUM_ROUNDS) + self._server_addr = ctx.get(Constant.RUNNER_CTX_SERVER_ADDR) + self._data_loader = ctx.get(Constant.RUNNER_CTX_DATA_LOADER) + self._tb_dir = ctx.get(Constant.RUNNER_CTX_TB_DIR) + self._model_dir = ctx.get(Constant.RUNNER_CTX_MODEL_DIR) + + if self.use_gpus: + # mapping each rank to a GPU (can set to cuda:0 if simulating with only one gpu) + self.logger.info(f"Training with GPU {self._rank}") + self.xgb_params["device"] = f"cuda:{self._rank}" + + self.logger.info(f"Using xgb params: {self.xgb_params}") + params = XGBoostParams( + xgb_params=self.xgb_params, + num_rounds=self._num_rounds, + early_stopping_rounds=self.early_stopping_rounds, + verbose_eval=self.verbose_eval, + ) + + self.logger.info(f"server address is {self._server_addr}") + communicator_env = { + "xgboost_communicator": "federated", + "federated_server_address": f"{self._server_addr}", + "federated_world_size": self._world_size, + "federated_rank": self._rank, + } + with xgb.collective.CommunicatorContext(**communicator_env): + # Load the data. Dmatrix must be created with column split mode in CommunicatorContext for vertical FL + train_data, val_data = self._data_loader.load_data(self._client_name) + + bst = self._xgb_train(params, train_data, val_data) + + # Save the model. + bst.save_model(os.path.join(self._model_dir, self.model_file_name)) + xgb.collective.communicator_print("Finished training\n") + + self._stopped = True + + def stop(self): + # currently no way to stop the runner + pass + + def is_stopped(self) -> (bool, int): + return self._stopped, 0 diff --git a/nvflare/app_common/xgb/runners/xgb_runner.py b/nvflare/app_common/xgb/runners/xgb_runner.py new file mode 100644 index 0000000000..0dae41ada6 --- /dev/null +++ b/nvflare/app_common/xgb/runners/xgb_runner.py @@ -0,0 +1,63 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from abc import ABC, abstractmethod + +from nvflare.apis.fl_context import FLContext + + +class XGBRunner(ABC): + + """An XGBRunner implements XGB (server or client) processing logic.""" + + def initialize(self, fl_ctx: FLContext): + """Called by Controller/Executor to initialize the runner. + This happens when the job is about to start. + + Args: + fl_ctx: FL context + + Returns: None + + """ + pass + + @abstractmethod + def run(self, ctx: dict): + """Called to start the execution of XGB processing logic. + + Args: + ctx: the contextual info to help the runner execution + + Returns: None + + """ + pass + + @abstractmethod + def stop(self): + """Called to stop the runner. + + Returns: + + """ + pass + + @abstractmethod + def is_stopped(self) -> (bool, int): + """Called to check whether the runner is already stopped. + + Returns: whether the runner is stopped. If stopped, the exit code. + + """ + pass diff --git a/nvflare/app_common/xgb/runners/xgb_server_runner.py b/nvflare/app_common/xgb/runners/xgb_server_runner.py new file mode 100644 index 0000000000..92a4f81c35 --- /dev/null +++ b/nvflare/app_common/xgb/runners/xgb_server_runner.py @@ -0,0 +1,42 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import xgboost.federated as xgb_federated + +from nvflare.app_common.xgb.defs import Constant +from nvflare.app_common.xgb.runners.xgb_runner import XGBRunner + + +class XGBServerRunner(XGBRunner): + def __init__(self): + self._port = None + self._world_size = None + self._stopped = False + + def run(self, ctx: dict): + self._port = ctx.get(Constant.RUNNER_CTX_PORT) + self._world_size = ctx.get(Constant.RUNNER_CTX_WORLD_SIZE) + + xgb_federated.run_federated_server( + port=self._port, + world_size=self._world_size, + ) + self._stopped = True + + def stop(self): + # no way to start currently + pass + + def is_stopped(self) -> (bool, int): + return self._stopped, 0 diff --git a/nvflare/app_common/xgb/sender.py b/nvflare/app_common/xgb/sender.py new file mode 100644 index 0000000000..7177fbb214 --- /dev/null +++ b/nvflare/app_common/xgb/sender.py @@ -0,0 +1,89 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from nvflare.apis.shareable import ReturnCode, Shareable +from nvflare.apis.signal import Signal +from nvflare.fuel.f3.cellnet.fqcn import FQCN +from nvflare.fuel.utils.obj_utils import get_logger + +from .defs import Constant + + +class Sender: + """ + A Sender is used to send XGB requests from the client to the server and wait for reply. + TBD: currently the sender simply sends the request with an aux message. It will be enhanced to be more + reliable in dealing with unstable network. + """ + + def __init__(self, engine, timeout): + """Constructor + + Args: + engine: the client engine that can send aux messages + timeout: the timeout for XGB requests + """ + self.engine = engine + self.timeout = timeout + self.logger = get_logger(self) + + def _extract_result(self, reply, expected_op): + if not reply: + return None + if not isinstance(reply, dict): + self.logger.error(f"expect reply to be a dict but got {type(reply)}") + return None + result = reply.get(FQCN.ROOT_SERVER) + if not result: + self.logger.error(f"no reply from {FQCN.ROOT_SERVER} for request {expected_op}") + return None + if not isinstance(result, Shareable): + self.logger.error(f"expect result to be a Shareable but got {type(result)}") + return None + rc = result.get_return_code() + if rc != ReturnCode.OK: + self.logger.error(f"server failed to process request: {rc=}") + return None + reply_op = result.get_header(Constant.MSG_KEY_XGB_OP) + if reply_op != expected_op: + self.logger.error(f"received op {reply_op} != expected op {expected_op}") + return None + return result + + def send_to_server(self, op: str, req: Shareable, abort_signal: Signal): + """Send an XGB request to the server. + + Args: + op: the XGB operation code + req: the XGB request + abort_signal: used for checking whether the job is aborted. + + Returns: reply from the server + + Note: when this method is enhanced to be more reliable, we'll keep resending until either the request is + sent successfully or the job is aborted. + + """ + req.set_header(Constant.MSG_KEY_XGB_OP, op) + + server_name = FQCN.ROOT_SERVER + with self.engine.new_context() as fl_ctx: + reply = self.engine.send_aux_request( + targets=[server_name], + topic=Constant.TOPIC_XGB_REQUEST, + request=req, + timeout=self.timeout, + fl_ctx=fl_ctx, + ) + return self._extract_result(reply, op) diff --git a/nvflare/app_common/xgb/tb.py b/nvflare/app_common/xgb/tb.py new file mode 100644 index 0000000000..0719d5b57d --- /dev/null +++ b/nvflare/app_common/xgb/tb.py @@ -0,0 +1,36 @@ +# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +import os + +import xgboost.callback + + +class TensorBoardCallback(xgboost.callback.TrainingCallback): + def __init__(self, app_dir: str, tensorboard): + xgboost.callback.TrainingCallback.__init__(self) + self.train_writer = tensorboard.SummaryWriter(log_dir=os.path.join(app_dir, "train-auc/")) + self.val_writer = tensorboard.SummaryWriter(log_dir=os.path.join(app_dir, "val-auc/")) + + def after_iteration(self, model, epoch: int, evals_log: xgboost.callback.TrainingCallback.EvalsLog): + if not evals_log: + return False + + for data, metric in evals_log.items(): + for metric_name, log in metric.items(): + score = log[-1][0] if isinstance(log[-1], tuple) else log[-1] + if data == "train": + self.train_writer.add_scalar(metric_name, score, epoch) + else: + self.val_writer.add_scalar(metric_name, score, epoch) + return False diff --git a/nvflare/app_common/xgb/xgb_params.py b/nvflare/app_common/xgb/xgb_params.py new file mode 100644 index 0000000000..bf5d4f9b81 --- /dev/null +++ b/nvflare/app_common/xgb/xgb_params.py @@ -0,0 +1,29 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +class XGBoostParams: + def __init__(self, xgb_params: dict, num_rounds=10, early_stopping_rounds=2, verbose_eval=False): + """Container for all XGBoost parameters. + + Args: + xgb_params: This dict is passed to `xgboost.train()` as the first argument `params`. + It contains all the Booster parameters. + Please refer to XGBoost documentation for details: + https://xgboost.readthedocs.io/en/stable/python/python_api.html#module-xgboost.training + """ + self.num_rounds = num_rounds + self.early_stopping_rounds = early_stopping_rounds + self.verbose_eval = verbose_eval + self.xgb_params: dict = xgb_params if xgb_params else {} From 1bc71968f1f815e158f2996969793fcead61081c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Wed, 7 Feb 2024 16:43:36 -0800 Subject: [PATCH 35/39] Separate send heartbeat out (#2356) --- .../executors/client_api_launcher_executor.py | 2 +- .../app_common/executors/launcher_executor.py | 2 +- .../app_common/executors/task_exchanger.py | 6 ++--- nvflare/fuel/utils/pipe/cell_pipe.py | 10 +++++++ nvflare/fuel/utils/pipe/file_pipe.py | 16 ++++++------ nvflare/fuel/utils/pipe/pipe.py | 11 +++++--- nvflare/fuel/utils/pipe/pipe_handler.py | 26 ++++++++++++++----- 7 files changed, 50 insertions(+), 23 deletions(-) diff --git a/nvflare/app_common/executors/client_api_launcher_executor.py b/nvflare/app_common/executors/client_api_launcher_executor.py index 24df1587b3..776c5a0206 100644 --- a/nvflare/app_common/executors/client_api_launcher_executor.py +++ b/nvflare/app_common/executors/client_api_launcher_executor.py @@ -33,7 +33,7 @@ def __init__( external_execution_wait: float = 5.0, peer_read_timeout: Optional[float] = None, monitor_interval: float = 0.01, - read_interval: float = 0.001, + read_interval: float = 0.5, heartbeat_interval: float = 5.0, heartbeat_timeout: float = 30.0, workers: int = 4, diff --git a/nvflare/app_common/executors/launcher_executor.py b/nvflare/app_common/executors/launcher_executor.py index a9fd1fab34..90cc6a3d0a 100644 --- a/nvflare/app_common/executors/launcher_executor.py +++ b/nvflare/app_common/executors/launcher_executor.py @@ -45,7 +45,7 @@ def __init__( external_execution_wait: float = 5.0, peer_read_timeout: Optional[float] = None, monitor_interval: float = 1.0, - read_interval: float = 0.1, + read_interval: float = 0.5, heartbeat_interval: float = 5.0, heartbeat_timeout: float = 30.0, workers: int = 1, diff --git a/nvflare/app_common/executors/task_exchanger.py b/nvflare/app_common/executors/task_exchanger.py index 663459317c..77a7b19bb9 100644 --- a/nvflare/app_common/executors/task_exchanger.py +++ b/nvflare/app_common/executors/task_exchanger.py @@ -33,7 +33,7 @@ class TaskExchanger(Executor): def __init__( self, pipe_id: str, - read_interval: float = 0.1, + read_interval: float = 0.5, heartbeat_interval: float = 5.0, heartbeat_timeout: Optional[float] = 30.0, resend_interval: float = 2.0, @@ -48,7 +48,7 @@ def __init__( Args: pipe_id (str): component id of pipe. read_interval (float): how often to read from pipe. - Defaults to 0.1. + Defaults to 0.5. heartbeat_interval (float): how often to send heartbeat to peer. Defaults to 5.0. heartbeat_timeout (float, optional): how long to wait for a @@ -115,7 +115,7 @@ def handle_event(self, event_type: str, fl_ctx: FLContext): self.pipe_handler.set_status_cb(self._pipe_status_cb) self.pipe.open(self.pipe_channel_name) self.pipe_handler.start() - elif event_type == EventType.END_RUN: + elif event_type == EventType.ABOUT_TO_END_RUN: self.log_info(fl_ctx, "Stopping pipe handler") if self.pipe_handler: self.pipe_handler.notify_end("end_of_job") diff --git a/nvflare/fuel/utils/pipe/cell_pipe.py b/nvflare/fuel/utils/pipe/cell_pipe.py index 9a0ceb7469..ab95ec437e 100644 --- a/nvflare/fuel/utils/pipe/cell_pipe.py +++ b/nvflare/fuel/utils/pipe/cell_pipe.py @@ -211,6 +211,16 @@ def set_cell_cb(self, channel_name: str): self.logger.info(f"registered CellPipe request CB for {self.channel}") def send(self, msg: Message, timeout=None) -> bool: + """Sends the specified message to the peer. + + Args: + msg: the message to be sent + timeout: if specified, number of secs to wait for the peer to read the message. + If not specified, wait indefinitely. + + Returns: + Whether the message is read by the peer. + """ with self.pipe_lock: if self.closed: raise BrokenPipeError("pipe closed") diff --git a/nvflare/fuel/utils/pipe/file_pipe.py b/nvflare/fuel/utils/pipe/file_pipe.py index 3d02df3dc9..d5d7384876 100644 --- a/nvflare/fuel/utils/pipe/file_pipe.py +++ b/nvflare/fuel/utils/pipe/file_pipe.py @@ -135,7 +135,7 @@ def clear(self): self._clear_dir(self.y_path) self._clear_dir(self.t_path) - def _monitor_file(self, file_path: str, timeout) -> bool: + def _monitor_file(self, file_path: str, timeout=None) -> bool: """Monitors the file until it's read-and-removed by peer, or timed out. If timeout, remove the file. @@ -147,8 +147,6 @@ def _monitor_file(self, file_path: str, timeout) -> bool: Returns: whether the file has been read and removed """ - if not timeout: - return False start = time.time() while True: if not self.pipe_path: @@ -156,7 +154,7 @@ def _monitor_file(self, file_path: str, timeout) -> bool: if not os.path.exists(file_path): return True - if time.time() - start > timeout: + if timeout and time.time() - start > timeout: # timed out - try to delete the file try: os.remove(file_path) @@ -247,13 +245,15 @@ def y_get(self, timeout=None): return self._get_from_dir(self.y_path, timeout) def send(self, msg: Message, timeout=None) -> bool: - """ + """Sends the specified message to the peer. Args: - msg: - timeout: + msg: the message to be sent + timeout: if specified, number of secs to wait for the peer to read the message. + If not specified, wait indefinitely. - Returns: whether the message is read by peer (if timeout is specified) + Returns: + Whether the message is read by the peer. """ if not self.pipe_path: diff --git a/nvflare/fuel/utils/pipe/pipe.py b/nvflare/fuel/utils/pipe/pipe.py index 5993896e7d..c4aeb81b3e 100644 --- a/nvflare/fuel/utils/pipe/pipe.py +++ b/nvflare/fuel/utils/pipe/pipe.py @@ -99,14 +99,15 @@ def clear(self): @abstractmethod def send(self, msg: Message, timeout=None) -> bool: - """Send the specified message to the peer. + """Sends the specified message to the peer. Args: msg: the message to be sent timeout: if specified, number of secs to wait for the peer to read the message. + If not specified, wait indefinitely. - Returns: whether the message is read by the peer. - If timeout is not specified, always return False. + Returns: + Whether the message is read by the peer. """ pass @@ -117,8 +118,10 @@ def receive(self, timeout=None) -> Union[None, Message]: Args: timeout: how long (number of seconds) to try + If not specified, return right away. - Returns: the message received; or None if no message + Returns: + the message received; or None if no message """ pass diff --git a/nvflare/fuel/utils/pipe/pipe_handler.py b/nvflare/fuel/utils/pipe/pipe_handler.py index 27ff51ac26..4826c0bfa4 100644 --- a/nvflare/fuel/utils/pipe/pipe_handler.py +++ b/nvflare/fuel/utils/pipe/pipe_handler.py @@ -107,6 +107,9 @@ def __init__( self.peer_is_up_or_dead = threading.Event() self._pause = False self._last_heartbeat_received_time = None + self._check_interval = 0.01 + self.heartbeat_sender = threading.Thread(target=self._heartbeat) + self.heartbeat_sender.daemon = True def set_status_cb(self, cb, *args, **kwargs): """Set CB for status handling. When the peer status is changed (ABORT, END, GONE), this CB is called. @@ -208,6 +211,9 @@ def start(self): if not self.reader.is_alive(): self.reader.start() + if not self.heartbeat_sender.is_alive(): + self.heartbeat_sender.start() + def stop(self, close_pipe=True): """Stops the handler and optionally close the monitored pipe. @@ -231,7 +237,7 @@ def send_to_peer(self, msg: Message, timeout=None, abort_signal: Signal = None) Args: msg: message to be sent timeout: how long to wait for the peer to read the data. - If not specified, return False immediately. + If not specified, will use ``self.default_request_timeout``. abort_signal: Returns: @@ -285,15 +291,13 @@ def _read(self): def _try_read(self): self._last_heartbeat_received_time = time.time() - last_heartbeat_sent_time = 0.0 while not self.asked_to_stop: - now = time.time() - if self._pause: time.sleep(self.read_interval) continue msg = self.pipe.receive() + now = time.time() if msg: self._last_heartbeat_received_time = now @@ -318,13 +322,23 @@ def _try_read(self): ) break + time.sleep(self.read_interval) + self.reader = None + + def _heartbeat(self): + last_heartbeat_sent_time = 0.0 + while not self.asked_to_stop: + if self._pause: + time.sleep(self._check_interval) + continue + now = time.time() + # send heartbeat to the peer if now - last_heartbeat_sent_time > self.heartbeat_interval: self.send_to_peer(self._make_event_message(Topic.HEARTBEAT, "")) last_heartbeat_sent_time = now - time.sleep(self.read_interval) - self.reader = None + time.sleep(self._check_interval) def get_next(self) -> Optional[Message]: """Gets the next message from the message queue. From 22405a3162be5178a5932e8afa8a9d00e1cedc53 Mon Sep 17 00:00:00 2001 From: Sean Yang Date: Thu, 8 Feb 2024 09:58:37 -0800 Subject: [PATCH 36/39] setup for tf load_data (#2360) --- .../data/test_configs/standalone_job/internal_tf.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/integration_test/data/test_configs/standalone_job/internal_tf.yml b/tests/integration_test/data/test_configs/standalone_job/internal_tf.yml index 38bfb1641e..5dbee25399 100644 --- a/tests/integration_test/data/test_configs/standalone_job/internal_tf.yml +++ b/tests/integration_test/data/test_configs/standalone_job/internal_tf.yml @@ -22,3 +22,5 @@ tests: "data": { "run_finished": True } validators: - path: tests.integration_test.src.validators.TFModelValidator + setup: + - python -c "import tensorflow as tf; tf.keras.datasets.mnist.load_data()" From 5341fc117a288d077c69b255df73799e009c75e4 Mon Sep 17 00:00:00 2001 From: Sean Yang Date: Thu, 15 Feb 2024 17:43:50 -0800 Subject: [PATCH 37/39] tf setup and teardown for integration tests (#2366) --- examples/nvflare_setup.ipynb | 2 +- .../test_configs/standalone_job/hello_tf_examples.yml | 8 ++++++++ .../data/test_configs/standalone_job/internal_tf.yml | 2 ++ 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/examples/nvflare_setup.ipynb b/examples/nvflare_setup.ipynb index eede563c67..c911037cca 100644 --- a/examples/nvflare_setup.ipynb +++ b/examples/nvflare_setup.ipynb @@ -23,7 +23,7 @@ "source": [ "### Install NVFLARE from PyPI\n", "```\n", - "pip install 'nvflare>=2.3.0'\n", + "pip install 'nvflare~=2.4.0'\n", "\n", "```\n", "We do not recommend running NVFlare CLI commands in jupyter notebook cells." diff --git a/tests/integration_test/data/test_configs/standalone_job/hello_tf_examples.yml b/tests/integration_test/data/test_configs/standalone_job/hello_tf_examples.yml index 7f104c77b0..39b1a59c3b 100644 --- a/tests/integration_test/data/test_configs/standalone_job/hello_tf_examples.yml +++ b/tests/integration_test/data/test_configs/standalone_job/hello_tf_examples.yml @@ -22,6 +22,10 @@ tests: "data": { "run_finished": True } validators: - path: tests.integration_test.src.validators.TFModelValidator + setup: + - python -c "import tensorflow as tf; tf.keras.datasets.mnist.load_data()" + teardown: + - rm ~/.keras/datasets/mnist.npz - test_name: "run hello-tf2" event_sequence: - "trigger": @@ -39,3 +43,7 @@ tests: "data": { "run_finished": True } validators: - path: tests.integration_test.src.validators.TFModelValidator + setup: + - python -c "import tensorflow as tf; tf.keras.datasets.mnist.load_data()" + teardown: + - rm ~/.keras/datasets/mnist.npz diff --git a/tests/integration_test/data/test_configs/standalone_job/internal_tf.yml b/tests/integration_test/data/test_configs/standalone_job/internal_tf.yml index 5dbee25399..59daaf8dbe 100644 --- a/tests/integration_test/data/test_configs/standalone_job/internal_tf.yml +++ b/tests/integration_test/data/test_configs/standalone_job/internal_tf.yml @@ -24,3 +24,5 @@ tests: - path: tests.integration_test.src.validators.TFModelValidator setup: - python -c "import tensorflow as tf; tf.keras.datasets.mnist.load_data()" + teardown: + - rm ~/.keras/datasets/mnist.npz From eaa46877c5291ffa7ba8181360cf149b1473eeaf Mon Sep 17 00:00:00 2001 From: Holger Roth <6304754+holgerroth@users.noreply.github.com> Date: Tue, 20 Feb 2024 11:02:25 -0500 Subject: [PATCH 38/39] fix nemo example link (#2373) --- integration/nemo/examples/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/integration/nemo/examples/README.md b/integration/nemo/examples/README.md index 7551091184..2e5bd21c2a 100644 --- a/integration/nemo/examples/README.md +++ b/integration/nemo/examples/README.md @@ -5,7 +5,7 @@ In this example, we utilize NeMo's [PEFT](https://docs.nvidia.com/deeplearning/n methods to showcase how to adapt a large language model (LLM) to a downstream task, such as financial sentiment predictions. -### [Supervised fine-tuning (SFT) with NeMo and NVFlare](./prompt_learning/README.md) +### [Supervised fine-tuning (SFT) with NeMo and NVFlare](./supervised_fine_tuning/README.md) An example of using [NVIDIA FLARE](https://nvflare.readthedocs.io/en/main/index.html) with NeMo for [supervised fine-tuning (SFT)](https://github.com/NVIDIA/NeMo-Megatron-Launcher#5152-sft-training) to fine-tune all parameters of a large language model (LLM) on supervised data to teach the model how to follow user specified instructions. From 3460970cdff8ec9710742a8bcf3c0ccd5c1d267a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Yuan-Ting=20Hsieh=20=28=E8=AC=9D=E6=B2=85=E5=BB=B7=29?= Date: Tue, 20 Feb 2024 13:08:37 -0800 Subject: [PATCH 39/39] Update outdated links (#2368) --- .github/workflows/markdown-links-check.yml | 3 --- .github/workflows/premerge.yml | 2 -- examples/advanced/cifar10/README.md | 2 +- examples/advanced/cifar10/cifar10-sim/README.md | 2 +- examples/advanced/nlp-ner/README.md | 2 +- examples/advanced/sklearn-kmeans/README.md | 2 +- examples/advanced/sklearn-linear/README.md | 2 +- examples/advanced/sklearn-svm/README.md | 4 ++-- .../vertical_federated_learning/cifar10-splitnn/README.md | 2 +- examples/tutorials/flare_simulator.ipynb | 2 +- 10 files changed, 9 insertions(+), 14 deletions(-) diff --git a/.github/workflows/markdown-links-check.yml b/.github/workflows/markdown-links-check.yml index 0f0c3f4505..186c14d178 100644 --- a/.github/workflows/markdown-links-check.yml +++ b/.github/workflows/markdown-links-check.yml @@ -17,10 +17,7 @@ name: Check Markdown links on: push: - branches: [ "main", "dev" ] pull_request: - # The branches below must be a subset of the branches above - branches: [ "main", "dev" ] jobs: markdown-link-check: diff --git a/.github/workflows/premerge.yml b/.github/workflows/premerge.yml index bc9048752a..932275df7e 100644 --- a/.github/workflows/premerge.yml +++ b/.github/workflows/premerge.yml @@ -17,8 +17,6 @@ name: pre-merge on: # quick tests for pull requests and the releasing branches push: - branches: - - dev pull_request: workflow_dispatch: diff --git a/examples/advanced/cifar10/README.md b/examples/advanced/cifar10/README.md index 1b641fb953..4e75b03e8b 100644 --- a/examples/advanced/cifar10/README.md +++ b/examples/advanced/cifar10/README.md @@ -6,7 +6,7 @@ Please make sure you set up virtual environment and follows [example root readme This example includes instructions on running [FedAvg](https://arxiv.org/abs/1602.05629), [FedProx](https://arxiv.org/abs/1812.06127), [FedOpt](https://arxiv.org/abs/2003.00295), and [SCAFFOLD](https://arxiv.org/abs/1910.06378) algorithms using NVFlare's -[FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/fl_simulator.html). +[FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/nvflare_cli/fl_simulator.html). ### [Real-world Federated Learning with CIFAR-10](./cifar10-real-world/README.md) Real-world FL deployment requires secure provisioning and the admin API to submit jobs. diff --git a/examples/advanced/cifar10/cifar10-sim/README.md b/examples/advanced/cifar10/cifar10-sim/README.md index 6feeaeb86e..230c7042e2 100644 --- a/examples/advanced/cifar10/cifar10-sim/README.md +++ b/examples/advanced/cifar10/cifar10-sim/README.md @@ -35,7 +35,7 @@ To speed up the following experiments, first download the [CIFAR-10](https://www ## 3. Run simulated FL experiments -We are using NVFlare's [FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/fl_simulator.html) to run the following experiments. +We are using NVFlare's [FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/nvflare_cli/fl_simulator.html) to run the following experiments. The output root of where to save the results is set in [./run_simulator.sh](./run_simulator.sh) as `RESULT_ROOT=/tmp/nvflare/sim_cifar10`. diff --git a/examples/advanced/nlp-ner/README.md b/examples/advanced/nlp-ner/README.md index 7ee52a3482..d110d944ba 100644 --- a/examples/advanced/nlp-ner/README.md +++ b/examples/advanced/nlp-ner/README.md @@ -52,7 +52,7 @@ Let's take a closer look at the word-label correspondence: As shown above, the task is to capture the keywords related to medical findings. ## Run automated experiments -We use the NVFlare [simulator](https://nvflare.readthedocs.io/en/latest/user_guide/fl_simulator.html) to run the FL training. +We use the NVFlare [simulator](https://nvflare.readthedocs.io/en/latest/user_guide/nvflare_cli/fl_simulator.html) to run the FL training. Set `PYTHONPATH` to include custom files of this example: ``` export PYTHONPATH=${PWD} diff --git a/examples/advanced/sklearn-kmeans/README.md b/examples/advanced/sklearn-kmeans/README.md index 5b826609bd..a3d7cdb158 100644 --- a/examples/advanced/sklearn-kmeans/README.md +++ b/examples/advanced/sklearn-kmeans/README.md @@ -117,7 +117,7 @@ Below is a sample config for site-1, saved to `./jobs/sklearn_kmeans_3_uniform/a ``` ## Run experiment with FL simulator -The [FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/fl_simulator.html) simulates FL experiments or debugging codes, +The [FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/nvflare_cli/fl_simulator.html) simulates FL experiments or debugging codes, not for real-world FL deployment. We can run the FL simulator with 3 clients under the uniform data split with ```commandline diff --git a/examples/advanced/sklearn-linear/README.md b/examples/advanced/sklearn-linear/README.md index 1284360101..54b9905fe9 100644 --- a/examples/advanced/sklearn-linear/README.md +++ b/examples/advanced/sklearn-linear/README.md @@ -101,7 +101,7 @@ Below is a sample config for site-1, saved to `./jobs/sklearn_linear_5_uniform/a ``` ## Run experiment with FL simulator -[FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/fl_simulator.html) is used to simulate FL experiments or debug codes, not for real FL deployment. +[FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/nvflare_cli/fl_simulator.html) is used to simulate FL experiments or debug codes, not for real FL deployment. We can run the FL simulator with five clients under the uniform data split with ```commandline bash run_experiment_simulator.sh diff --git a/examples/advanced/sklearn-svm/README.md b/examples/advanced/sklearn-svm/README.md index b2263ea90f..fc17102bd8 100644 --- a/examples/advanced/sklearn-svm/README.md +++ b/examples/advanced/sklearn-svm/README.md @@ -34,7 +34,7 @@ Under this setting, federated learning can be formulated in two steps: Unlike other iterative federated algorithms, federated SVM only involves these two training steps. Hence, in the server config, we have -```json +``` "num_rounds": 2 ``` The first round is the training round, performing local training and global aggregation. @@ -116,7 +116,7 @@ Below is a sample config for site-1, saved to `./jobs/sklearn_svm_3_uniform/app_ ``` ## Run experiment with FL simulator -[FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/fl_simulator.html) is used to simulate FL experiments or debug codes, not for real FL deployment. +[FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/nvflare_cli/fl_simulator.html) is used to simulate FL experiments or debug codes, not for real FL deployment. We can run the FL simulator with three clients under the uniform data split with ```commandline bash run_experiment_simulator.sh diff --git a/examples/advanced/vertical_federated_learning/cifar10-splitnn/README.md b/examples/advanced/vertical_federated_learning/cifar10-splitnn/README.md index 181d104baa..02caf835d4 100644 --- a/examples/advanced/vertical_federated_learning/cifar10-splitnn/README.md +++ b/examples/advanced/vertical_federated_learning/cifar10-splitnn/README.md @@ -1,7 +1,7 @@ # Split Learning with CIFAR-10 This example includes instructions on how to run [split learning](https://arxiv.org/abs/1810.06060) (SL) using the -CIFAR-10 dataset and the [FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/fl_simulator.html). +CIFAR-10 dataset and the [FL simulator](https://nvflare.readthedocs.io/en/latest/user_guide/nvflare_cli/fl_simulator.html). We assume one client holds the images, and the other client holds the labels to compute losses and accuracy metrics. Activations and corresponding gradients are being exchanged between the clients using NVFlare. diff --git a/examples/tutorials/flare_simulator.ipynb b/examples/tutorials/flare_simulator.ipynb index dd58e77901..89aa73df0f 100644 --- a/examples/tutorials/flare_simulator.ipynb +++ b/examples/tutorials/flare_simulator.ipynb @@ -7,7 +7,7 @@ "source": [ "## Intro to the FL Simulator\n", "\n", - "The [FL Simulator](https://nvflare.readthedocs.io/en/main/user_guide/fl_simulator.html) runs a local simulation of a running NVFLARE FL deployment. This allows researchers to test and debug an application without provisioning a real, distributed FL project. The FL Simulator runs a server and multiple clients in the same local process, with communication that mimics a real deployment. This allows researchers to more quickly build out new components and jobs that can be directly used in a production deployment.\n", + "The [FL Simulator](https://nvflare.readthedocs.io/en/latest/user_guide/nvflare_cli/fl_simulator.html) runs a local simulation of a running NVFLARE FL deployment. This allows researchers to test and debug an application without provisioning a real, distributed FL project. The FL Simulator runs a server and multiple clients in the same local process, with communication that mimics a real deployment. This allows researchers to more quickly build out new components and jobs that can be directly used in a production deployment.\n", "\n", "### Setup\n", "The NVFlare [Getting Started Guide](https://nvflare.readthedocs.io/en/main/getting_started.html) provides instructions for setting up FLARE on a local system or in a Docker image. We've also cloned the NVFlare GitHub in our top-level working directory."