Add docs sections and rearrangements [skip ci] (#2047)

* add for HUB

* reorganize docs

* updates

* address PR comments

* address PR comments

* update examples for docs

---------

Co-authored-by: Yuan-Ting Hsieh (謝沅廷) <[email protected]>

docs/example_applications_algorithms.rst

@@ -23,58 +23,66 @@ NVIDIA FLARE has several tutorials and examples to help you get started with fed
 
 The following tutorials and quickstart guides walk you through some of these examples:
 
-  1. **Hello World Examples** which can be run from the `hello_world notebook <https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/hello_world.ipynb>`_.
+  1. **Hello World** introduction to NVFlare.
 
-     1.1. Workflows
+     1.1. Deep Learning to Federated Learning
+          * `Deep Learning to Federated Learning (GitHub) <https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/ml-to-fl>`_ - Example for converting Deep Learning (DL) to Federated Learning (FL).
+
+     1.2. Step-by-Step Examples
+          * `Step-by-Step Examples (GitHub) <https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/step-by-step>`__ - Step-by-step examples for running a federated learning project with NVFlare.
+
+  2. **Hello World Examples** which can be run from the `hello_world notebook <https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/hello_world.ipynb>`_.
+
+     2.1. Workflows
           * :ref:`Hello Scatter and Gather <hello_scatter_and_gather>` - Example using the Scatter And Gather (SAG) workflow with a Numpy trainer
           * :ref:`Hello Cross-Site Validation <hello_cross_val>` - Example using the Cross Site Model Eval workflow with a Numpy trainer
           * `Hello Cyclic Weight Transfer (GitHub) <https://github.com/NVIDIA/NVFlare/tree/main/examples/hello-cyclic>`_ - Example using the CyclicController workflow to implement `Cyclic Weight Transfer <https://pubmed.ncbi.nlm.nih.gov/29617797/>`_ with TensorFlow as the deep learning training framework
 
-     1.2. Deep Learning
+     2.2. Deep Learning
           * :ref:`Hello PyTorch <hello_pt>` - Example image classifier using FedAvg and PyTorch as the deep learning training framework
           * :ref:`Hello TensorFlow <hello_tf2>` - Example image classifier using FedAvg and TensorFlow as the deep learning training frameworks
 
-  2. **Tutorial notebooks**
+  3. **Tutorial notebooks**
 
     * `Intro to the FL Simulator <https://github.com/NVIDIA/NVFlare/blob/main/examples/tutorials/flare_simulator.ipynb>`_ - Shows how to use the :ref:`fl_simulator` to run a local simulation of an NVFLARE deployment to test and debug an application without provisioning a real FL project.
     * `Hello FLARE API <https://github.com/NVIDIA/NVFlare/blob/main/examples/tutorials/flare_api.ipynb>`_ - Goes through the different commands of the :ref:`flare_api` to show the syntax and usage of each.
     * `NVFLARE in POC Mode <https://github.com/NVIDIA/NVFlare/blob/main/examples/tutorials/setup_poc.ipynb>`_ - Shows how to use :ref:`POC mode <poc_command>` to test the features of a full FLARE deployment on a single machine.
 
-  3. **FL algorithms**
+  4. **FL algorithms**
 
     * `Federated Learning with CIFAR-10 (GitHub) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/cifar10>`_ - Includes examples of using FedAvg, FedProx, FedOpt, SCAFFOLD, homomorphic encryption, and streaming of TensorBoard metrics to the server during training
     * :ref:`Federated XGBoost <federated_xgboost>` - Includes examples of histogram-based and tree-based algorithms. Tree-based algorithms also includes bagging and cyclic approaches
 
-  4. **Traditional ML examples**
+  5. **Traditional ML examples**
 
     * `Federated Linear Model with Scikit-learn (GitHub) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/sklearn-linear>`_ - For an example of using NVIDIA FLARE with `scikit-learn <https://scikit-learn.org/>`_, a widely used open-source machine learning library that supports supervised and unsupervised learning.
     * `Federated K-Means Clustering with Scikit-learn (GitHub) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/sklearn-kmeans>`_ - NVIDIA FLARE with `scikit-learn <https://scikit-learn.org/>`_ and k-Means.
     * `Federated SVM with Scikit-learn (GitHub) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/sklearn-svm>`_ - NVIDIA FLARE with `scikit-learn <https://scikit-learn.org/>`_ and `SVM <https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html>`_.
     * `Federated Learning for Random Forest based on XGBoost (GitHub) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/random_forest>`_ - Example of using NVIDIA FLARE with `scikit-learn <https://scikit-learn.org/>`_ and `Random Forest <https://xgboost.readthedocs.io/en/stable/tutorials/rf.html>`_.
 
-  5. **Medical Image Analysis**
+  6. **Medical Image Analysis**
 
     * `MONAI Integration (GitHub) <https://github.com/NVIDIA/NVFlare/tree/main/integration/monai>`_ - For an example of using NVIDIA FLARE to train a 3D medical image analysis model using federated averaging (FedAvg) and MONAI Bundle `MONAI <https://monai.io/>`_
     * `Federated Learning with Differential Privacy for BraTS18 segmentation (GitHub) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/brats18>`_ - Illustrates the use of differential privacy for training brain tumor segmentation models using federated learning
     * `Federated Learning for Prostate Segmentation from Multi-source Data (GitHub) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/prostate>`_ - Example of training a multi-institutional prostate segmentation model using `FedAvg <https://arxiv.org/abs/1602.05629>`_, `FedProx <https://arxiv.org/abs/1812.06127>`_, and `Ditto <https://arxiv.org/abs/2012.04221>`_
 
-  6. **Federated Statistics**
+  7. **Federated Statistics**
 
     * :ref:`Federated Statistic Overview <federated_statistics>` - Discuss the overall federated statistics features
     * `Federated Statistics for medical imaging (Github) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/federated-statistics/image_stats/README.md>`_ - Example of gathering local image histogram to compute the global dataset histograms.
     * `Federated Statistics for tabular data with DataFrame (Github) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/federated-statistics/df_stats/README.md>`_ - Example of gathering local statistics summary from Pandas DataFrame to compute the global dataset statistics.
     * `Federated Statistics with Monai Statistics integration for Spleen CT Image (Github) <https://github.com/NVIDIA/NVFlare/tree/main/integration/monai/examples/spleen_ct_segmentation/README.md>`_ - Example demonstrated Monai statistics integration and few other features in federated statistics
 
-  7. **Federated Site Policies**
+  8. **Federated Site Policies**
 
     * `Federated Policies (Github) <https://github.com/NVIDIA/NVFlare/blob/main/examples/advanced/federated-policies/README.rst>`_ - Discuss the federated site policies for authorization, resource and data privacy management
 
-  8. **Experiment tracking**
+  9. **Experiment tracking**
 
     * :ref:`FL Experiment Tracking with TensorBoard Streaming <tensorboard_streaming>` - Example building on Hello PyTorch with TensorBoard streaming from clients to server
     * :ref:`FL Experiment Tracking with MLflow <experiment_tracking_mlflow>` - Example integrating Hello PyTorch with MLflow with streaming from clients to server
 
-  9. **NLP**
+  10. **NLP**
 
     * `NLP-NER (Github) <https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/nlp-ner/README.md>`_ - Illustrates both `BERT <https://github.com/google-research/bert>`_ and `GPT-2 <https://github.com/openai/gpt-2>`_ models from `Hugging Face <https://huggingface.co/>`_ (`BERT-base-uncased <https://huggingface.co/bert-base-uncased>`_, `GPT-2 <https://huggingface.co/gpt2>`_) on a Named Entity Recognition (NER) task using the `NCBI disease dataset <https://pubmed.ncbi.nlm.nih.gov/24393765/>`_.
 

docs/examples/hello_world_examples.rst

@@ -5,8 +5,11 @@ Hello World examples can be run from the `hello_world notebook <https://github.c
 
 .. toctree::
 
+   Deep Learning to Federated Learning (GitHub) <https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/ml-to-fl>
+   Step-by-Step Examples (GitHub) <https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/step-by-step>
    hello_scatter_and_gather
    hello_cross_val
    Hello Cyclic Weight Transfer (GitHub) <https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/hello-cyclic>
    hello_pt
    hello_tf2
+   Hello Client Controlled Workflow (GitHub) <https://github.com/NVIDIA/NVFlare/blob/main/examples/hello-world/hello-ccwf>

docs/faq.rst

@@ -181,22 +181,22 @@ Operational
 
 #. What is the difference between the Admin client and the FL client?
 
-    The :ref:`Admin client <operating_nvflare>` is used to control the state of the server's controller workflow and only interacts with the
+    The :ref:`FLARE Console <operating_nvflare>` is used to control the state of the server's controller workflow and only interacts with the
     server.  FL clients poll the server and perform tasks based on the state of the server.  The Admin client does not
     interact directly with FL client.
 
 #. Where does the Admin client run?
 
-    The :ref:`Admin client <operating_nvflare>` runs as a standalone process, typically on a researcher's workstation or laptop.
+    The :ref:`FLARE Console <operating_nvflare>` runs as a standalone process, typically on a researcher's workstation or laptop.
 
 #. What can you do with the Admin client?
 
-    The :ref:`Admin client <operating_nvflare>` is used to orchestrate the FL study, including starting and stopping server
+    The :ref:`FLARE Console <operating_nvflare>` is used to orchestrate the FL study, including starting and stopping server
     and clients, deploying applications, and managing FL experiments.
 
 #. How can I get the global model at the end of training? What can I do to resolve keys not matching with the model defined?
 
-    You can use the download_job command with the :ref:`Admin client <operating_nvflare>` to get the job result into the admin
+    You can use the download_job command with the :ref:`FLARE Console <operating_nvflare>` to get the job result into the admin
     transfer folder. The model is saved in a dict depending on the persistor you used, so you might need to access it with
     ``model.load_state_dict(torch.load(path_to_model)["model"])`` if you used PTFileModelPersistor because
     PTModelPersistenceFormatManager saves the model under the key "model".

docs/programming_guide.rst

@@ -41,7 +41,6 @@ Please refer to :ref:`application` for more details.
    programming_guide/data_exchange_object
    programming_guide/fl_context
    programming_guide/fl_component
-   programming_guide/serialization
    programming_guide/filters
    programming_guide/event_system
    programming_guide/component_configuration

docs/programming_guide/serialization.rst

@@ -1,7 +1 @@
-.. _serialization:
-
-Serialization
-=============
-
-Due to security concerns, `pickle <https://docs.python.org/3/library/pickle.html>` has been replaced with FOBS (Flare object serialization) in NVFlare to exchange data between the server and clients.
-See `<https://github.com/NVIDIA/NVFlare/blob/main/nvflare/fuel/utils/fobs/README.rst>`_ for usage guidelines.
+See :ref:`serialization`.

docs/programming_guide/system_architecture.rst

@@ -15,7 +15,7 @@ Concepts and System Components
 
 Spec-based Programming for System Service Objects
 =================================================
-NVIDIA FLARE 2.1.0 needs additional services to implement the HA feature:
+NVIDIA FLARE needs additional services to implement the HA feature:
 storage, overseer, job definition management, etc. There are many ways to implement such services. For example,
 storage could be implemented with a file system, AWS S3, or some database technologies. Similarly, job definition
 management could be done with simple file reading or a sophisticated solution with a database or search engine.
@@ -34,13 +34,13 @@ See the example :ref:`project_yml` for how these components are configured in St
 
 Overseer
 --------
-The Overseer is a system component newly introduced in 2.1.0 that determines the hot FL server at any time for high availability.
+The Overseer is a system component that determines the hot FL server at any time for high availability.
 The name of the Overseer must be unique and in the format of fully qualified domain names.  During
 provisioning time, if the name is specified incorrectly, either being duplicate or containing incompatible
 characters, the provision command will fail with an error message. It is possible to use a unique hostname rather than
 FQDN, with the IP mapped to the hostname by having it added to ``/etc/hosts``.
 
-NVIDIA FLARE 2.1.0 comes with HTTPS-based overseer.  Users are welcome to change the name and port arguments of the overseer
+NVIDIA FLARE comes with an HTTPS-based overseer.  Users are welcome to change the name and port arguments of the overseer
 in project.yml to fit their deployment environment.
 
 The Overseer will receive a Startup kit, which includes the start.sh shell script, its certificate and private key,
@@ -66,7 +66,7 @@ their own Overseer Agent.
 NVIDIA FLARE provides two implementations:
 
     - :class:`HttpOverseerAgent<nvflare.ha.overseer_agent.HttpOverseerAgent>` to work with the Overseer server. For NVIDIA
-      FLARE 2.1.0, the provisioning tool will automatically map parameters specified in Overseer into the arguments for
+      FLARE, the provisioning tool will automatically map parameters specified in Overseer into the arguments for
       the HttpOverseerAgent.
     - :class:`DummyOverseerAgent<nvflare.ha.dummy_overseer_agent.DummyOverseerAgent>` is a dummy agent that simply
       returns the configured endpoint as the hot FL server. The dummy agent is used when a single FL server is configured

docs/real_world_fl.rst

@@ -14,6 +14,8 @@ help gather information to provision a project and distribute startup kits, see
 For more details on what you can do with apps with custom components and
 the flexibility that the Controller and Worker APIs bring, see the :ref:`programming_guide`.
 
+For setting up authorization policies, see :ref:`federated authorization <federated_authorization>`.
+
 You can also see some `example applications <https://github.com/NVIDIA/NVFlare/tree/main/examples>`_ integrating with
 `Clara Train <https://docs.nvidia.com/clara/clara-train-sdk/>`_ and
 `MONAI <https://github.com/Project-MONAI/tutorials/tree/master/federated_learning/nvflare>`_

docs/user_guide.rst

@@ -22,10 +22,9 @@ which are explained in more detail in their own sections linked below.
    user_guide/dashboard_api
    user_guide/dashboard_ui
    user_guide/nvflare_security
-   user_guide/federated_authorization
    user_guide/site_policy_management
-   user_guide/authorization_policy_previewer
    user_guide/docker_compose
    user_guide/helm_chart
    user_guide/logging_configuration
    user_guide/confidential_computing
+   user_guide/hierarchy_unification_bridge

docs/user_guide/authorization_policy_previewer.rst

@@ -1,63 +1 @@
-.. _authorization_policy_previewer:
-
-******************************
-Authorization Policy Previewer
-******************************
-
-Authorization is an important security feature of NVFLARE. In NVFLARE 2.2, each site defines its own authorization policy. Since authorization policy is vital for system security, and many people can now define policies, it's important to be able to validate the policies before deploying them to production.
-
-The Authorization Policy Previewer is a tool for validating authorization policy definitions. The tool provides an interactive user interface and commands for the user to validate different aspects of policy definitions:
-
-    - Show defined roles and rights
-    - Show the content of the policy definition
-    - Show the permission matrix (role/right/conditions)
-    - Evaluate a right against a specified user
-
-Start Authorization Policy Previewer
-======================================
-To start the Authorization Policy Previewer, enter this command on a terminal:
-
-.. code-block:: shell
-
-  nvflare authz_preview -p <authorization_policy_file>
-
-The authorization_policy_file must be a JSON file that follows authorization file format.
-
-If the file is not a valid JSON file or does not follow authorization file format, this command will exit with exception.
-
-Execute Authorization Policy Previewer Commands
-================================================
-If the Authorization Policy Previewer is successfully started, the prompt ">" will be displayed and for command input.
-
-To get the complete list of commands, enter "?" on the prompt.
-
-Most commands are self-explanatory, except for the "eval_right". With this command, you can evaluate a specified right against a specified user (name:org:role) to make sure the result is correct.
-
-Role Rights
-===========
-Most permissions in the policy file may be defined with Command Categories. However, once the policy file is loaded, categories are already resolved to individual commands, following the fallback mechanism.
-
-Use the ``show_role_rights command`` to verify that all commands have the right permissions for all roles.
-
-Evaluate a Right
-================
-The syntax of the ``eval_right`` command is:
-
-.. code-block:: shell
-
-  eval_right site_org right_name user_name:org:role [submitter_name:org:role]
-
-where:
-
-.. code-block::
-
-    site_org - the organization of the site
-    right_name - the right to be evaluated. You can use the "show_rights" command to list all available commands.
-    User specification - a user spec has three pieces of information separated by colons. Name is the name of the user; org is the organization that the user belongs to; and role is the user's role. You can use the "show_roles" command to list all available roles.
-    Submitter specification - some job related commands can evaluate the relation between the user and the submitter of a job. Submitter spec has the same format as user spec.
-
-Please refer to :ref:`Federated Authorization <federated_authorization>` for details on the right definition and evaluation.
-
-Stop Authorization Policy Previewer
-======================================
-To exit from the Authorization Policy Previewer, enter the "bye" command at the prompt.
+See :ref:`authorization_policy_previewer`.