diff --git a/.gitattributes b/.gitattributes index 2bd1bcf..12ec4fe 100644 --- a/.gitattributes +++ b/.gitattributes @@ -1 +1 @@ -docker_image/nsx-t-install-03252019-ua.tar filter=lfs diff=lfs merge=lfs -text +docker_image/nsx-t-install-05202019-ua.tar filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md index 78ce7bc..a3300c0 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,11 @@ - + # nsx-t-datacenter-ci-pipelines This repository provides an easy-to-use automation framework that installs and configures NSX-T on vCenter clusters where PKS and/or PAS can be deployed. ## Overview -Under the hood, there is a Concourse pipeline which is to be set up by a Docker container which the user creates. The Concourse pipeline is in turn run in three Docker containers: DB, worker, and web container. +Under the hood, there is a Concourse pipeline which is to be set up by a Docker container which the user creates. The Concourse pipeline is in turn run in three Docker containers: DB, worker, and web container. The Concourse pipeline performs the following jobs: 1. Deploy NSX manager, controllers and edges; @@ -18,7 +18,7 @@ See the wiki: https://github.com/vmware/nsx-t-datacenter-ci-pipelines/wiki ## Try it out On a Ubuntu VM with at least ~20GB of space, ``` -wget https://github.com/vmware/nsx-t-datacenter-ci-pipelines/raw/master/docker_image/nsx-t-install-09122018.tar -O nsx-t-install.tar +wget https://github.com/vmware/nsx-t-datacenter-ci-pipelines/raw/master/docker_image/nsx-t-install-05202019-ua.tar -O nsx-t-install.tar docker load -i nsx-t-install.tar mkdir -p /home/concourse ``` @@ -27,18 +27,34 @@ Create nsx_pipeline_config.yml based on a sample config file, e.g. https://githu docker run --name nsx-t-install -d \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /home/concourse:/home/concourse \ - -e CONCOURSE_URL="http://10.33.75.99:8080" \ - -e EXTERNAL_DNS="10.33.38.1" \ + -e CONCOURSE_URL='http://:8080' \ + -e EXTERNAL_DNS='' \ -e IMAGE_WEBSERVER_PORT=40001 \ -e VMWARE_USER='' \ -e VMWARE_PASSWORD='' \ nsx-t-install ``` -Set CONCOURSE_URL to http://:8080 (host_ip is the IP address of the primary NIC of the VM running the container (example: 10.85.99.130); it is not the loopback address. Set EXTERNAL_DNS to the DNS server (it should be able to resolve the vCenter hostname, and public names e.g. github.com), and IMAGE_WEBSERVER_PORT to the port number provided in the nsx_pipeline_config.yml parameter nsx_image_webserver (recommendation: 40001). +Set CONCOURSE_URL to http://:8080 (host_ip is the IP address of the primary NIC of the VM running the container (example: 10.85.99.130); it is not the loopback address. Set EXTERNAL_DNS to the DNS server (it should be able to resolve the vCenter hostname, and public names e.g. github.com), and IMAGE_WEBSERVER_PORT to the port number provided in the nsx_pipeline_config.yml parameter nsx_image_webserver (recommendation: 40001). -The above command will automatically download the ovftool (e.g. VMware-ovftool-4.3.0-xxxxxxx-lin.x86_64.bundle) and NSX OVA (nsx-unified-appliance-2.2.0.0.0.xxxxxxx.ova) files from myvmware.com. If you have already downloaded the two files manually, place them under /home/concourse, and run above command with VMWARE_USER and VMWARE_PASSWORD skipped. +The above command will automatically download the ovftool (e.g. VMware-ovftool-4.3.0-xxxxxxx-lin.x86_64.bundle) and NSX OVA (nsx-unified-appliance-2.4.0.0.0.xxxxxxx.ova) files from myvmware.com. If you have already downloaded the two files manually, place them under /home/concourse, and you can run above command with VMWARE_USER and VMWARE_PASSWORD skipped. By default, the docker image from master/nsxt_2.4.0 branch downloads nsx ova version 2.4.0. If deploying earlier version (e.g. NSX-T 2.3.0), simply add `` -e NSXT_VERSION=2.3.0 `` in the docker run command above, or use the docker image from nsxt_2.3.0 branch. -Browse to the Concourse pipeline: http:///teams/main/pipelines/install-nsx-t/ (example: http://10.85.99.130:8080/teams/main/pipelines/install-nsx-t/) and click on the plus on the upper right corner to trigger a build to install NSX-T. +--- +__If running the pipeline on existing concourse environment and not using the nsx-t-install image, please perform following additional steps:__ in nsx_pipeline_config.yml that was created under /home/concourse, add the following lines at the beginning: +``` +nsxt_ansible_branch=master +nsx_t_pipeline_branch=master +``` +if deploying NSX-T 2.4.0 or later, or +``` +nsxt_ansible_branch=v1.0.0 +nsx_t_pipeline_branch=nsxt_2.3.0 +``` +if deploying NSX-T 2.3.0 or earlier. +Also, if ovftool and ova files were downloaded manually, add ``ova_file_name=`` and ``ovftool_file_name=`` in nsx_pipeline_config.yml as well. + +--- + +Browse to the Concourse pipeline: http:///teams/main/pipelines/install-nsx-t/ (example: http://10.85.99.130:8080/teams/main/pipelines/install-nsx-t/) and click on the plus on the upper right corner to trigger a build to install NSX-T. If you are prompted with a username and password, use 'nsx' and 'vmware'. Check out the [Troubleshooting Guide](https://github.com/vmware/nsx-t-datacenter-ci-pipelines/wiki/Troubleshooting) for troubleshooting tips. @@ -61,5 +77,3 @@ so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - - diff --git a/docker_image/Dockerfile b/docker_image/Dockerfile index ef9e344..b16b2f5 100644 --- a/docker_image/Dockerfile +++ b/docker_image/Dockerfile @@ -1,9 +1,9 @@ -FROM ubuntu:17.10 +FROM ubuntu:18.04 ADD run.sh /home/run.sh # https://docs.docker.com/compose/install/#install-compose RUN apt-get update && \ - apt-get install -y curl openssh-client git wget && \ + apt-get install -y vim curl openssh-client git wget && \ curl -sSL https://get.docker.com/ | sh && \ curl -L --fail https://github.com/docker/compose/releases/download/1.23.2/run.sh -o /usr/local/bin/docker-compose && \ chmod +x /usr/local/bin/docker-compose && \ diff --git a/docker_image/nsx-t-install-05202019-ua.tar b/docker_image/nsx-t-install-05202019-ua.tar new file mode 100644 index 0000000..3a5c81b --- /dev/null +++ b/docker_image/nsx-t-install-05202019-ua.tar @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:aee9fddfd8d38a2ff182f1e93075c811441730c3dca3f88dc182cb4ca7654a0b +size 546060800 diff --git a/docker_image/run.sh b/docker_image/run.sh index f7899e1..6974d07 100755 --- a/docker_image/run.sh +++ b/docker_image/run.sh @@ -80,16 +80,20 @@ docker run --name nginx-server -v ${BIND_MOUNT_DIR}:/usr/share/nginx/html:ro -p mkdir -p $ROOT_WORK_DIR cd $ROOT_WORK_DIR -git clone https://github.com/concourse/concourse-docker.git -git clone -b $nsx_t_pipeline_branch --single-branch https://github.com/vmware/nsx-t-datacenter-ci-pipelines.git -concourse_docker_dir=${ROOT_WORK_DIR}/concourse-docker +#git clone https://github.com/concourse/concourse-docker.git +#concourse_docker_dir=${ROOT_WORK_DIR}/concourse-docker +#cp ${concourse_docker_dir}/keys/generate $BIND_MOUNT_DIR +#./generate + +git clone -b $nsx_t_pipeline_branch --single-branch https://github.com/vmware/nsx-t-datacenter-ci-pipelines.git pipeline_dir=${ROOT_WORK_DIR}/nsx-t-datacenter-ci-pipelines -cp ${concourse_docker_dir}/keys/generate $BIND_MOUNT_DIR cp ${pipeline_dir}/docker_compose/docker-compose.yml $BIND_MOUNT_DIR +cp ${pipeline_dir}/functions/generate-keys.sh $BIND_MOUNT_DIR cd $BIND_MOUNT_DIR -./generate +chmod +x generate-keys.sh +./generate-keys.sh # prepare the yaml for docker compose concourse_version=4.2.1 diff --git a/functions/generate-keys.sh b/functions/generate-keys.sh new file mode 100644 index 0000000..82d3694 --- /dev/null +++ b/functions/generate-keys.sh @@ -0,0 +1,18 @@ +#!/usr/bin/env bash + +set -e -u -x + +mkdir -p keys/web keys/worker + +openssl genpkey -algorithm RSA -out ./keys/web/tsa_host_key -pkeyopt rsa_keygen_bits:4096 +openssl genpkey -algorithm RSA -out ./keys/web/session_signing_key -pkeyopt rsa_keygen_bits:4096 +openssl genpkey -algorithm RSA -out ./keys/worker/worker_key -pkeyopt rsa_keygen_bits:4096 + +chmod 600 ./keys/ -R + +ssh-keygen -y -f ./keys/web/tsa_host_key > ./keys/web/tsa_host_key.pub +ssh-keygen -y -f ./keys/web/session_signing_key > ./keys/web/session_signing_key.pub +ssh-keygen -y -f ./keys/worker/worker_key > ./keys/worker/worker_key.pub + +cp ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys +cp ./keys/web/tsa_host_key.pub ./keys/worker \ No newline at end of file diff --git a/nsxt_yaml/basic_resources.yml b/nsxt_yaml/basic_resources.yml index 03f849e..58f46d5 100644 --- a/nsxt_yaml/basic_resources.yml +++ b/nsxt_yaml/basic_resources.yml @@ -153,7 +153,8 @@ node_deployment_info: resource_type: "HostNode" display_name: "{{hostvars[item].fabric_node_name}}" - ip_addresses: "{{hostvars[item].ip}}" + ip_addresses: + - "{{hostvars[item].ip}}" os_type: "ESXI" os_version: "{{hostvars[item].esx_os_version}}" host_credential: diff --git a/nsxt_yaml/basic_topology.yml b/nsxt_yaml/basic_topology.yml index d85dc49..ed7f153 100644 --- a/nsxt_yaml/basic_topology.yml +++ b/nsxt_yaml/basic_topology.yml @@ -329,7 +329,7 @@ type: UplinkHostSwitchProfile host_switch_name: "{{overlay_host_switch}}" pnics: - - device_name: fp-eth1 + - device_name: fp-eth0 uplink_name: "{{uplink_1_name}}" ip_assignment_spec: resource_type: StaticIpPoolSpec @@ -339,7 +339,7 @@ type: UplinkHostSwitchProfile host_switch_name: "{{hostvars[item].vlan_logical_switch_name}}_hostswitch" pnics: - - device_name: fp-eth0 + - device_name: fp-eth1 uplink_name: "{{uplink_1_name}}" - host_switch_profiles: - name: "{{edge_uplink_prof}}" @@ -366,9 +366,8 @@ placement_type: VsphereDeploymentConfig vc_name: "{{compute_manager_name}}" data_network_ids: - - "{{hostvars[item].vc_uplink_network_for_edge}}" - "{{hostvars[item].vc_overlay_network_for_edge}}" - - "{{hostvars[item].vc_management_network_for_edge}}" + - "{{hostvars[item].vc_uplink_network_for_edge}}" management_network_id: "{{hostvars[item].vc_management_network_for_edge}}" hostname: "{{hostvars[item].hostname}}" compute_id: "{{hostvars[item].vc_cluster_for_edge}}" diff --git a/pipelines/nsx-t-install.yml b/pipelines/nsx-t-install.yml index 2289573..a89c823 100644 --- a/pipelines/nsx-t-install.yml +++ b/pipelines/nsx-t-install.yml @@ -20,6 +20,7 @@ nsx_t_gen_params: &nsx-t-gen-params unified_appliance_int: ((unified_appliance)) nsx_manager_ips_int: ((nsx_manager_ips)) + nsx_manager_virtual_ip_int: ((nsx_manager_virtual_ip)) nsx_manager_username_int: ((nsx_manager_username)) nsx_manager_password_int: ((nsx_manager_password)) nsx_manager_hostname_prefix_int: ((nsx_manager_hostname_prefix)) diff --git a/sample_parameters/PAS_and_PKS/nsx_pipeline_config.yml b/sample_parameters/PAS_and_PKS/nsx_pipeline_config.yml index 644be28..5c5fa7c 100644 --- a/sample_parameters/PAS_and_PKS/nsx_pipeline_config.yml +++ b/sample_parameters/PAS_and_PKS/nsx_pipeline_config.yml @@ -13,6 +13,7 @@ netmask: 255.255.255.0 # NSX manager cluster configs nsx_manager_ips: 192.168.110.33,192.168.110.34,192.168.110.35 # 3 node cluster is recommended. 1 is minimum, 3 is max +nsx_manager_virtual_ip: 0.0.0.0 nsx_manager_username: admin nsx_manager_password: Admin!23Admin nsx_manager_hostname_prefix: "nsxt-mgr" # Generated hostname: nsxt-mgr-1.corp.local.io, will be FQDN diff --git a/sample_parameters/PAS_only/nsx_pipeline_config.yml b/sample_parameters/PAS_only/nsx_pipeline_config.yml index 26e9c8e..ab16abf 100644 --- a/sample_parameters/PAS_only/nsx_pipeline_config.yml +++ b/sample_parameters/PAS_only/nsx_pipeline_config.yml @@ -13,6 +13,7 @@ netmask: 255.255.255.0 # NSX manager cluster configs nsx_manager_ips: 192.168.110.33,192.168.110.34,192.168.110.35 # 3 node cluster is recommended. 1 is minimum, 3 is max +nsx_manager_virtual_ip: 0.0.0.0 nsx_manager_username: admin nsx_manager_password: Admin!23Admin nsx_manager_hostname_prefix: "nsxt-mgr" # Generated hostname: nsxt-mgr-1.corp.local.io, will be FQDN diff --git a/sample_parameters/PKS_only/nsx_pipeline_config.yml b/sample_parameters/PKS_only/nsx_pipeline_config.yml index 3e2600b..49d8eb9 100644 --- a/sample_parameters/PKS_only/nsx_pipeline_config.yml +++ b/sample_parameters/PKS_only/nsx_pipeline_config.yml @@ -13,6 +13,7 @@ netmask: 255.255.255.0 # NSX manager cluster configs nsx_manager_ips: 192.168.110.33,192.168.110.34,192.168.110.35 # 3 node cluster is recommended. 1 is minimum, 3 is max +nsx_manager_virtual_ip: 0.0.0.0 nsx_manager_username: admin nsx_manager_password: Admin!23Admin nsx_manager_hostname_prefix: "nsxt-mgr" # Generated hostname: nsxt-mgr-1.corp.local.io, will be FQDN diff --git a/sample_parameters/raw/nsx.yml b/sample_parameters/raw/nsx.yml index 02c40a2..3e85f60 100644 --- a/sample_parameters/raw/nsx.yml +++ b/sample_parameters/raw/nsx.yml @@ -13,6 +13,7 @@ netmask: 255.255.255.0 # NSX manager cluster configs nsx_manager_ips: 192.168.110.33,192.168.110.34,192.168.110.35 # 3 node cluster is recommended. 1 is minimum, 3 is max +nsx_manager_virtual_ip: 0.0.0.0 nsx_manager_username: admin nsx_manager_password: Admin!23Admin nsx_manager_hostname_prefix: "nsxt-mgr" # Generated hostname: nsxt-mgr-1.corp.local.io, will be FQDN diff --git a/tasks/add-nsx-t-routers/task.yml b/tasks/add-nsx-t-routers/task.yml index 29524ed..a54e6f7 100644 --- a/tasks/add-nsx-t-routers/task.yml +++ b/tasks/add-nsx-t-routers/task.yml @@ -4,7 +4,7 @@ platform: linux image_resource: type: docker-image - source: {repository: nsxedgegen/nsx-t-gen-worker} + source: {repository: dyanngg/nsx-t-gen-worker} inputs: - name: nsx-t-gen-pipeline diff --git a/tasks/config-nsx-t-extras/nsx_t_gen.py b/tasks/config-nsx-t-extras/nsx_t_gen.py index aae3f7c..b9c82c2 100755 --- a/tasks/config-nsx-t-extras/nsx_t_gen.py +++ b/tasks/config-nsx-t-extras/nsx_t_gen.py @@ -23,11 +23,13 @@ TRUST_MGMT_CRLS_ENDPOINT = '%s%s' % (API_VERSION, '/trust-management/crls') TRUST_MGMT_SELF_SIGN_CERT = '%s%s' % (API_VERSION, '/trust-management/csrs/') TRUST_MGMT_UPDATE_CERT = '%s%s' % (API_VERSION, '/node/services/http?action=apply_certificate') +CLUSTER_CERT_UPDATE_ENDPOINT = '%s%s' % (API_VERSION, '/cluster/api-certificate?action=set_cluster_certificate') LBR_SERVICES_ENDPOINT = '%s%s' % (API_VERSION, '/loadbalancer/services') LBR_VIRTUAL_SERVER_ENDPOINT = '%s%s' % (API_VERSION, '/loadbalancer/virtual-servers') LBR_POOLS_ENDPOINT = '%s%s' % (API_VERSION, '/loadbalancer/pools') LBR_MONITORS_ENDPOINT = '%s%s' % (API_VERSION, '/loadbalancer/monitors') NSGROUP_ENDPOINT = '%s%s' % (API_VERSION, '/ns-groups') +VIP_ENDPOINT = '%s%s' % (API_VERSION, '/cluster/api-virtual-ip') LBR_APPLICATION_PROFILE_ENDPOINT = '%s%s' % (API_VERSION, '/loadbalancer/application-profiles') LBR_PERSISTENCE_PROFILE_ENDPOINT = '%s%s' % (API_VERSION, '/loadbalancer/persistence-profiles') @@ -789,10 +791,14 @@ def does_comman_name_match(attr_list): update_api_endpint = '%s%s%s' % (TRUST_MGMT_UPDATE_CERT, '&certificate_id=', self_sign_csr_id) update_csr_response = client.post(update_api_endpint, '') - print('NSX Mgr updated to use newly generated CSR!!' + '\n Update response code:{}'.format(update_csr_response.status_code)) + cluster_cert_api_point = '%s%s%s' % (CLUSTER_CERT_UPDATE_ENDPOINT, '&certificate_id=', self_sign_csr_id) + cluster_cert_response = client.post(cluster_cert_api_point, '') + print('NSX Mgr cluster updated to use newly generated CSR!!' + + '\n Update response code:{}'.format(cluster_cert_response.status_code)) + ############################## # T0 Route redistribution @@ -1623,6 +1629,23 @@ def create_emtpy_ns_group(): return ns_group['id'] +def set_cluster_vip_address(): + vip_addr = os.getenv('nsx_manager_virtual_ip_int', '').strip() + if vip_addr == '' or vip_addr == 'null': + print('No yaml payload set for the NSX_T_LBR_SPEC, ignoring loadbalancer section!') + return + + current_vip = client.get(VIP_ENDPOINT).json()['ip_address'] + if current_vip != vip_addr: + clear_vip_endpoint = '%s?%s' % (VIP_ENDPOINT, 'action=clear_virtual_ip') + client.post(clear_vip_endpoint, None) + print('Setting new nsx manager virtual IP address at %s' % vip_addr) + new_vip_endpoint = '%s?%s&%s=%s' % (VIP_ENDPOINT, 'action=set_virtual_ip', 'ip_address', vip_addr) + client.post(new_vip_endpoint, None) + else: + print('Detected no change with nsx manager VIP address!') + + def get_args(): parser = argparse.ArgumentParser( description='Arguments for NSX resource config') @@ -1657,6 +1680,8 @@ def main(): load_ip_blocks() load_ip_pools() + set_cluster_vip_address() + # No support for switching profile in the ansible script yet # So create directly create_ha_switching_profile() diff --git a/tasks/config-nsx-t-extras/task.yml b/tasks/config-nsx-t-extras/task.yml index 12212bd..48bd38c 100755 --- a/tasks/config-nsx-t-extras/task.yml +++ b/tasks/config-nsx-t-extras/task.yml @@ -4,7 +4,7 @@ platform: linux image_resource: type: docker-image - source: {repository: nsxedgegen/nsx-t-gen-worker} + source: {repository: dyanngg/nsx-t-gen-worker} params: NSX_T_INSTALLER: diff --git a/tasks/install-nsx-t/task.yml b/tasks/install-nsx-t/task.yml index 7e5873d..6e4bb79 100755 --- a/tasks/install-nsx-t/task.yml +++ b/tasks/install-nsx-t/task.yml @@ -4,7 +4,7 @@ platform: linux image_resource: type: docker-image - source: {repository: nsxedgegen/nsx-t-gen-worker} + source: {repository: dyanngg/nsx-t-gen-worker} inputs: - name: nsx-t-gen-pipeline