Merge branch 'master' into multi_t0

Signed-off-by: Yang Ding <[email protected]>

.gitattributes

@@ -1 +1 @@
-docker_image/nsx-t-install-03252019-ua.tar filter=lfs diff=lfs merge=lfs -text
+docker_image/nsx-t-install-05202019-ua.tar filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,11 +1,11 @@
- 
+
 
 
 # nsx-t-datacenter-ci-pipelines
 This repository provides an easy-to-use automation framework that installs and configures NSX-T on vCenter clusters where PKS and/or PAS can be deployed.
 
 ## Overview
-Under the hood, there is a Concourse pipeline which is to be set up by a Docker container which the user creates. The Concourse pipeline is in turn run in three Docker containers: DB, worker, and web container. 
+Under the hood, there is a Concourse pipeline which is to be set up by a Docker container which the user creates. The Concourse pipeline is in turn run in three Docker containers: DB, worker, and web container.
 
 The Concourse pipeline performs the following jobs:
 1. Deploy NSX manager, controllers and edges;
@@ -18,7 +18,7 @@ See the wiki: https://github.com/vmware/nsx-t-datacenter-ci-pipelines/wiki
 ## Try it out
 On a Ubuntu VM with at least ~20GB of space,
 ```
-wget https://github.com/vmware/nsx-t-datacenter-ci-pipelines/raw/master/docker_image/nsx-t-install-09122018.tar -O nsx-t-install.tar
+wget https://github.com/vmware/nsx-t-datacenter-ci-pipelines/raw/master/docker_image/nsx-t-install-05202019-ua.tar -O nsx-t-install.tar
 docker load -i nsx-t-install.tar
 mkdir -p /home/concourse
 ```
@@ -27,18 +27,34 @@ Create nsx_pipeline_config.yml based on a sample config file, e.g. https://githu
 docker run --name nsx-t-install -d \
   -v /var/run/docker.sock:/var/run/docker.sock \
   -v /home/concourse:/home/concourse \
-  -e CONCOURSE_URL="http://10.33.75.99:8080" \
-  -e EXTERNAL_DNS="10.33.38.1" \
+  -e CONCOURSE_URL='http://<host_ip>:8080' \
+  -e EXTERNAL_DNS='<dns_server>' \
   -e IMAGE_WEBSERVER_PORT=40001 \
   -e VMWARE_USER='<myvmware_user_email>' \
   -e VMWARE_PASSWORD='<myvmware_password>' \
   nsx-t-install
 ```
-Set CONCOURSE_URL to http://<host_ip>:8080 (host_ip is the IP address of the primary NIC of the VM running the container (example: 10.85.99.130); it is not the loopback address. Set EXTERNAL_DNS to the DNS server (it should be able to resolve the vCenter hostname, and public names e.g. github.com), and  IMAGE_WEBSERVER_PORT to the port number provided in the  nsx_pipeline_config.yml parameter nsx_image_webserver (recommendation: 40001).
+Set CONCOURSE_URL to http://<host_ip>:8080 (host_ip is the IP address of the primary NIC of the VM running the container (example: 10.85.99.130); it is not the loopback address. Set EXTERNAL_DNS to the DNS server (it should be able to resolve the vCenter hostname, and public names e.g. github.com), and IMAGE_WEBSERVER_PORT to the port number provided in the nsx_pipeline_config.yml parameter nsx_image_webserver (recommendation: 40001).
 
-The above command will automatically download the ovftool (e.g. VMware-ovftool-4.3.0-xxxxxxx-lin.x86_64.bundle) and NSX OVA (nsx-unified-appliance-2.2.0.0.0.xxxxxxx.ova) files from myvmware.com. If you have already downloaded the two files manually, place them under /home/concourse, and run above command with VMWARE_USER and VMWARE_PASSWORD skipped.
+The above command will automatically download the ovftool (e.g. VMware-ovftool-4.3.0-xxxxxxx-lin.x86_64.bundle) and NSX OVA (nsx-unified-appliance-2.4.0.0.0.xxxxxxx.ova) files from myvmware.com. If you have already downloaded the two files manually, place them under /home/concourse, and you can run above command with VMWARE_USER and VMWARE_PASSWORD skipped. By default, the docker image from master/nsxt_2.4.0 branch downloads nsx ova version 2.4.0. If deploying earlier version (e.g. NSX-T 2.3.0), simply add `` -e NSXT_VERSION=2.3.0 `` in the docker run command above, or use the docker image from nsxt_2.3.0 branch.
 
-Browse to the Concourse pipeline: http://<CONCOURSE_URL>/teams/main/pipelines/install-nsx-t/ (example: http://10.85.99.130:8080/teams/main/pipelines/install-nsx-t/) and click on the plus on the upper right corner to trigger a build to install NSX-T.
+---
+__If running the pipeline on existing concourse environment and not using the nsx-t-install image, please perform following additional steps:__ in nsx_pipeline_config.yml that was created under /home/concourse, add the following lines at the beginning:
+```
+nsxt_ansible_branch=master
+nsx_t_pipeline_branch=master
+```
+if deploying NSX-T 2.4.0 or later, or
+```
+nsxt_ansible_branch=v1.0.0
+nsx_t_pipeline_branch=nsxt_2.3.0
+```
+if deploying NSX-T 2.3.0 or earlier.
+Also, if ovftool and ova files were downloaded manually, add ``ova_file_name=<ova_file_name>`` and ``ovftool_file_name=<ovftool_file_name>`` in nsx_pipeline_config.yml as well.
+
+---
+
+Browse to the Concourse pipeline: http://<CONCOURSE_URL>/teams/main/pipelines/install-nsx-t/ (example: http://10.85.99.130:8080/teams/main/pipelines/install-nsx-t/) and click on the plus on the upper right corner to trigger a build to install NSX-T. If you are prompted with a username and password, use 'nsx' and 'vmware'.
 
 Check out the [Troubleshooting Guide](https://github.com/vmware/nsx-t-datacenter-ci-pipelines/wiki/Troubleshooting) for troubleshooting tips.
 
@@ -61,5 +77,3 @@ so, subject to the following conditions:
 The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-

docker_image/Dockerfile

@@ -1,9 +1,9 @@
-FROM ubuntu:17.10
+FROM ubuntu:18.04
 ADD run.sh /home/run.sh
 
 # https://docs.docker.com/compose/install/#install-compose
 RUN apt-get update && \
-    apt-get install -y curl openssh-client git wget && \
+    apt-get install -y vim curl openssh-client git wget && \
     curl -sSL https://get.docker.com/ | sh && \
     curl -L --fail https://github.com/docker/compose/releases/download/1.23.2/run.sh -o /usr/local/bin/docker-compose && \
     chmod +x /usr/local/bin/docker-compose && \

docker_image/run.sh

@@ -80,16 +80,20 @@ docker run --name nginx-server -v ${BIND_MOUNT_DIR}:/usr/share/nginx/html:ro -p
 
 mkdir -p $ROOT_WORK_DIR
 cd $ROOT_WORK_DIR
-git clone https://github.com/concourse/concourse-docker.git
-git clone -b $nsx_t_pipeline_branch --single-branch https://github.com/vmware/nsx-t-datacenter-ci-pipelines.git
 
-concourse_docker_dir=${ROOT_WORK_DIR}/concourse-docker
+#git clone https://github.com/concourse/concourse-docker.git
+#concourse_docker_dir=${ROOT_WORK_DIR}/concourse-docker
+#cp ${concourse_docker_dir}/keys/generate $BIND_MOUNT_DIR
+#./generate
+
+git clone -b $nsx_t_pipeline_branch --single-branch https://github.com/vmware/nsx-t-datacenter-ci-pipelines.git
 pipeline_dir=${ROOT_WORK_DIR}/nsx-t-datacenter-ci-pipelines
-cp ${concourse_docker_dir}/keys/generate $BIND_MOUNT_DIR
 cp ${pipeline_dir}/docker_compose/docker-compose.yml $BIND_MOUNT_DIR
+cp ${pipeline_dir}/functions/generate-keys.sh $BIND_MOUNT_DIR
 
 cd $BIND_MOUNT_DIR
-./generate
+chmod +x generate-keys.sh
+./generate-keys.sh
 
 # prepare the yaml for docker compose
 concourse_version=4.2.1

functions/generate-keys.sh

@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+
+set -e -u -x
+
+mkdir -p keys/web keys/worker
+
+openssl genpkey -algorithm RSA -out ./keys/web/tsa_host_key -pkeyopt rsa_keygen_bits:4096
+openssl genpkey -algorithm RSA -out ./keys/web/session_signing_key -pkeyopt rsa_keygen_bits:4096
+openssl genpkey -algorithm RSA -out ./keys/worker/worker_key -pkeyopt rsa_keygen_bits:4096
+
+chmod 600 ./keys/ -R
+
+ssh-keygen -y -f  ./keys/web/tsa_host_key > ./keys/web/tsa_host_key.pub
+ssh-keygen -y -f  ./keys/web/session_signing_key > ./keys/web/session_signing_key.pub
+ssh-keygen -y -f  ./keys/worker/worker_key > ./keys/worker/worker_key.pub
+
+cp ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys
+cp ./keys/web/tsa_host_key.pub ./keys/worker

nsxt_yaml/basic_resources.yml

@@ -153,7 +153,8 @@
         node_deployment_info:
           resource_type: "HostNode"
           display_name: "{{hostvars[item].fabric_node_name}}"
-          ip_addresses: "{{hostvars[item].ip}}"
+          ip_addresses:
+          - "{{hostvars[item].ip}}"
           os_type: "ESXI"
           os_version: "{{hostvars[item].esx_os_version}}"
           host_credential:

nsxt_yaml/basic_topology.yml

@@ -329,7 +329,7 @@
               type: UplinkHostSwitchProfile
             host_switch_name: "{{overlay_host_switch}}"
             pnics:
-            - device_name: fp-eth1
+            - device_name: fp-eth0
               uplink_name: "{{uplink_1_name}}"
             ip_assignment_spec:
               resource_type: StaticIpPoolSpec
@@ -339,7 +339,7 @@
               type: UplinkHostSwitchProfile
             host_switch_name: "{{hostvars[item].vlan_logical_switch_name}}_hostswitch"
             pnics:
-            - device_name: fp-eth0
+            - device_name: fp-eth1
               uplink_name: "{{uplink_1_name}}"
           - host_switch_profiles:
             - name: "{{edge_uplink_prof}}"
@@ -366,9 +366,8 @@
               placement_type: VsphereDeploymentConfig
               vc_name: "{{compute_manager_name}}"
               data_network_ids:
-              - "{{hostvars[item].vc_uplink_network_for_edge}}"
               - "{{hostvars[item].vc_overlay_network_for_edge}}"
-              - "{{hostvars[item].vc_management_network_for_edge}}"
+              - "{{hostvars[item].vc_uplink_network_for_edge}}"
               management_network_id: "{{hostvars[item].vc_management_network_for_edge}}"
               hostname: "{{hostvars[item].hostname}}"
               compute_id: "{{hostvars[item].vc_cluster_for_edge}}"

pipelines/nsx-t-install.yml

@@ -20,6 +20,7 @@ nsx_t_gen_params: &nsx-t-gen-params
 
   unified_appliance_int: ((unified_appliance))
   nsx_manager_ips_int: ((nsx_manager_ips))
+  nsx_manager_virtual_ip_int: ((nsx_manager_virtual_ip))
   nsx_manager_username_int: ((nsx_manager_username))
   nsx_manager_password_int: ((nsx_manager_password))
   nsx_manager_hostname_prefix_int: ((nsx_manager_hostname_prefix))

sample_parameters/PAS_and_PKS/nsx_pipeline_config.yml

@@ -13,6 +13,7 @@ netmask: 255.255.255.0
 
 # NSX manager cluster configs
 nsx_manager_ips: 192.168.110.33,192.168.110.34,192.168.110.35 # 3 node cluster is recommended. 1 is minimum, 3 is max
+nsx_manager_virtual_ip: 0.0.0.0
 nsx_manager_username: admin
 nsx_manager_password: Admin!23Admin
 nsx_manager_hostname_prefix: "nsxt-mgr" # Generated hostname: nsxt-mgr-1.corp.local.io, will be FQDN

sample_parameters/PAS_only/nsx_pipeline_config.yml

@@ -13,6 +13,7 @@ netmask: 255.255.255.0
 
 # NSX manager cluster configs
 nsx_manager_ips: 192.168.110.33,192.168.110.34,192.168.110.35 # 3 node cluster is recommended. 1 is minimum, 3 is max
+nsx_manager_virtual_ip: 0.0.0.0
 nsx_manager_username: admin
 nsx_manager_password: Admin!23Admin
 nsx_manager_hostname_prefix: "nsxt-mgr" # Generated hostname: nsxt-mgr-1.corp.local.io, will be FQDN

sample_parameters/PKS_only/nsx_pipeline_config.yml

@@ -13,6 +13,7 @@ netmask: 255.255.255.0
 
 # NSX manager cluster configs
 nsx_manager_ips: 192.168.110.33,192.168.110.34,192.168.110.35 # 3 node cluster is recommended. 1 is minimum, 3 is max
+nsx_manager_virtual_ip: 0.0.0.0
 nsx_manager_username: admin
 nsx_manager_password: Admin!23Admin
 nsx_manager_hostname_prefix: "nsxt-mgr" # Generated hostname: nsxt-mgr-1.corp.local.io, will be FQDN

sample_parameters/raw/nsx.yml

@@ -13,6 +13,7 @@ netmask: 255.255.255.0
 
 # NSX manager cluster configs
 nsx_manager_ips: 192.168.110.33,192.168.110.34,192.168.110.35 # 3 node cluster is recommended. 1 is minimum, 3 is max
+nsx_manager_virtual_ip: 0.0.0.0
 nsx_manager_username: admin
 nsx_manager_password: Admin!23Admin
 nsx_manager_hostname_prefix: "nsxt-mgr" # Generated hostname: nsxt-mgr-1.corp.local.io, will be FQDN

tasks/add-nsx-t-routers/task.yml

@@ -4,7 +4,7 @@ platform: linux
 
 image_resource:
   type: docker-image
-  source: {repository: nsxedgegen/nsx-t-gen-worker}
+  source: {repository: dyanngg/nsx-t-gen-worker}
 
 inputs:
   - name: nsx-t-gen-pipeline

tasks/config-nsx-t-extras/nsx_t_gen.py

@@ -23,11 +23,13 @@
 TRUST_MGMT_CRLS_ENDPOINT     = '%s%s' % (API_VERSION, '/trust-management/crls')
 TRUST_MGMT_SELF_SIGN_CERT    = '%s%s' % (API_VERSION, '/trust-management/csrs/')
 TRUST_MGMT_UPDATE_CERT       = '%s%s' % (API_VERSION, '/node/services/http?action=apply_certificate')
+CLUSTER_CERT_UPDATE_ENDPOINT = '%s%s' % (API_VERSION, '/cluster/api-certificate?action=set_cluster_certificate')
 LBR_SERVICES_ENDPOINT        = '%s%s' % (API_VERSION, '/loadbalancer/services')
 LBR_VIRTUAL_SERVER_ENDPOINT  = '%s%s' % (API_VERSION, '/loadbalancer/virtual-servers')
 LBR_POOLS_ENDPOINT           = '%s%s' % (API_VERSION, '/loadbalancer/pools')
 LBR_MONITORS_ENDPOINT        = '%s%s' % (API_VERSION, '/loadbalancer/monitors')
 NSGROUP_ENDPOINT             = '%s%s' % (API_VERSION, '/ns-groups')
+VIP_ENDPOINT                 = '%s%s' % (API_VERSION, '/cluster/api-virtual-ip')
 
 LBR_APPLICATION_PROFILE_ENDPOINT = '%s%s' % (API_VERSION, '/loadbalancer/application-profiles')
 LBR_PERSISTENCE_PROFILE_ENDPOINT = '%s%s' % (API_VERSION, '/loadbalancer/persistence-profiles')
@@ -789,10 +791,14 @@ def does_comman_name_match(attr_list):
 
     update_api_endpint = '%s%s%s' % (TRUST_MGMT_UPDATE_CERT, '&certificate_id=', self_sign_csr_id)
     update_csr_response = client.post(update_api_endpint, '')
-
     print('NSX Mgr updated to use newly generated CSR!!'
           + '\n    Update response code:{}'.format(update_csr_response.status_code))
 
+    cluster_cert_api_point = '%s%s%s' % (CLUSTER_CERT_UPDATE_ENDPOINT, '&certificate_id=', self_sign_csr_id)
+    cluster_cert_response = client.post(cluster_cert_api_point, '')
+    print('NSX Mgr cluster updated to use newly generated CSR!!'
+          + '\n    Update response code:{}'.format(cluster_cert_response.status_code))
+
 
 ##############################
 # T0 Route redistribution
@@ -1623,6 +1629,23 @@ def create_emtpy_ns_group():
             return ns_group['id']
 
 
+def set_cluster_vip_address():
+    vip_addr = os.getenv('nsx_manager_virtual_ip_int', '').strip()
+    if vip_addr == '' or vip_addr == 'null':
+        print('No yaml payload set for the NSX_T_LBR_SPEC, ignoring loadbalancer section!')
+        return
+
+    current_vip = client.get(VIP_ENDPOINT).json()['ip_address']
+    if current_vip != vip_addr:
+        clear_vip_endpoint = '%s?%s' % (VIP_ENDPOINT, 'action=clear_virtual_ip')
+        client.post(clear_vip_endpoint, None)
+        print('Setting new nsx manager virtual IP address at %s' % vip_addr)
+        new_vip_endpoint = '%s?%s&%s=%s' % (VIP_ENDPOINT, 'action=set_virtual_ip', 'ip_address', vip_addr)
+        client.post(new_vip_endpoint, None)
+    else:
+        print('Detected no change with nsx manager VIP address!')
+
+
 def get_args():
     parser = argparse.ArgumentParser(
         description='Arguments for NSX resource config')
@@ -1657,6 +1680,8 @@ def main():
         load_ip_blocks()
         load_ip_pools()
 
+        set_cluster_vip_address()
+
         # No support for switching profile in the ansible script yet
         # So create directly
         create_ha_switching_profile()

tasks/config-nsx-t-extras/task.yml

@@ -4,7 +4,7 @@ platform: linux
 
 image_resource:
   type: docker-image
-  source: {repository: nsxedgegen/nsx-t-gen-worker}
+  source: {repository: dyanngg/nsx-t-gen-worker}
 
 params:
   NSX_T_INSTALLER:

tasks/install-nsx-t/task.yml

@@ -4,7 +4,7 @@ platform: linux
 
 image_resource:
   type: docker-image
-  source: {repository: nsxedgegen/nsx-t-gen-worker}
+  source: {repository: dyanngg/nsx-t-gen-worker}
 
 inputs:
   - name: nsx-t-gen-pipeline