diff --git a/docs/lesson01-setting-work-envs/01-create-aws-account.qmd b/docs/lesson01-setting-work-envs/01-create-aws-account.qmd index 6870189..d8ec735 100644 --- a/docs/lesson01-setting-work-envs/01-create-aws-account.qmd +++ b/docs/lesson01-setting-work-envs/01-create-aws-account.qmd @@ -1,23 +1,30 @@ --- title: "Create Your AWS Account" --- - ::: callout-warning ## Prerequisites +**Please read [Workshops Organisation](/index.qmd#workshops-organisation)** if you haven't done so. + +If you are attending a workshop using a **Cloud-SPAN AWS account**, you **don't need** to complete this episode. + +If you are self-studying the course **or** attending a workshop using **your AWS account**, you must complete this episode and for that you will need: -**Please read [Workshops Organisation](/index.qmd#course-overview)** if you haven't done so. To complete this episode you will need: - if you are self-studying the course **or** attending a workshop using **your AWS account**: - an email address - a credit card --- new accounts get one-year of AWS Free Tier but a card number must be entered on creating an account - the phone number associated with the credit card - the address associated with the credit card - if you are attending a workshop using a **Cloud-SPAN AWS account** (and an AWS Linux instance), you **don't need** to complete this episode. +- an email address +- a credit card --- new accounts get one-year of AWS Free Tier but a card number must be entered on creating an account +- the phone number associated with the credit card +- the address associated with the credit card ::: ## Introduction These are the main steps you will follow to open your AWS account: -1. Sign-up to AWS with your email (as username) and password. -2. Select your account type (Personal) and enter your contact information. -3. Enter Billing Information: your credit card details. -4. Confirm your identity to AWS through a phone call or SMS message. -5. Select support plan (Basic) and complete sign-up. -6. Login to your AWS account. +1. [Sign-up to AWS with your email (as username) and password](#sign-up-to-aws-with-your-email-and-password). +2. [Select your account type (Personal) and enter your contact information](#select-your-account-type-personal-and-enter-your-contact-information). +3. [Enter Billing Information: your credit card details](#enter-billing-information-your-credit-card-details). +4. [Confirm your identity to AWS through a phone call or SMS message](#confirm-your-identity-to-aws-through-a-phone-call-or-sms-message). +5. [Select support plan (Basic) and complete sign-up](#select-support-plan-basic-and-complete-sign-up). +6. [Login to your AWS account](#login-to-your-aws-account). ## 1. Sign-up to AWS with your email and password @@ -111,7 +118,7 @@ You will then be prompted to: 3. At this stage you can only login to the Root user account. In the next episode you will create an IAM account which you will use to create and manage AWS resources. ::: -Finally you will be logged in as Root user to the **Console Home** screen shown below, able to use your account. We will first configure your account in the next episode. +Finally you will be logged in as the Root user to the **Console Home** screen shown below, able to use your account. We will first configure your account in the next episode. ![.](/images/open-acc/using-acc04-signedin-options-services-and-cookies.PNG.jpg){width="900px" fig-alt="Screen shot of AWS (management) Console in the browser showing the first page after you have logged in"} diff --git a/docs/lesson01-setting-work-envs/02-configure-account.qmd b/docs/lesson01-setting-work-envs/02-configure-account.qmd index 58c2554..b4659c8 100644 --- a/docs/lesson01-setting-work-envs/02-configure-account.qmd +++ b/docs/lesson01-setting-work-envs/02-configure-account.qmd @@ -5,32 +5,34 @@ title: "Configure Your AWS Account" ::: callout-warning ## Prerequisites -**Please read [Workshops Organisation](/index.qmd#course-overview)** if you haven't done so. To complete this episode you will need: +**Please read [Workshops Organisation](/index.qmd#workshops-organisation)** if you haven't done so. -- if you are self-studying the course **or** attending a workshop using **your AWS account**: - - to have opened your AWS account as described in the first episode of this lesson: [Create Your AWS Account](/docs/lesson01-setting-work-envs/01-create-aws-account.qmd). - - to be logged in to your AWS account as the **Root User** (described also in that episode, at the end). - - ideally, your mobile phone to add multi-factor authentication (**MFA**) to your AWS account. However, **if you don't have a mobile phone, you can skip this step** and still use your AWS account. -- if you are attending a workshop using a **Cloud-SPAN AWS account** (and and AWS Linux instance), you **don't need** to complete this episode. +If you are attending a workshop using a **Cloud-SPAN AWS account**, you **don't need** to complete this episode. + +If you are self-studying the course **or** attending a workshop using **your AWS account**, you must complete this episode and for that you will need: + +- to have opened your AWS account as described in the first episode of this lesson: [Create Your AWS Account](/docs/lesson01-setting-work-envs/01-create-aws-account.qmd). +- to be logged in to your AWS account as the **Root User** (described also in that episode, at the end). +- ideally, your mobile phone to add multi-factor authentication (**MFA**) to your AWS account. However, **if you don't have a mobile phone, you can skip this step** and still use your AWS account. ::: ## Introduction These are the main steps you will follow to configure your AWS account: -1. **Change the default region of your account to Ireland.**\ +1. [Change the default region of your account to Ireland](#change-the-default-region-of-your-account-to-ireland).\ AWS services are provided through many regions around the world and a region is allocated by default. You will need to change the region of your account to Ireland because the Amazon Machine Image from which you will create your AWS instance is stored in the Ireland region. But you can later change your account region if you wish. -2. **Secure your AWS Root User account.**\ +2. [Secure your AWS Root User account](#secure-your-aws-root-user-account).\ The account your created in the last episode is your Root user account and can perform any operation including closing the account. It is best practice to use the Root user account only for high-level administration and to create the first **IAM** (Identity Access Managment) user account for day-to-day work and account management. It is also best practice to secure the Root user account with multi-factor authentication (MFA). -3. **Create an IAM user account to create and manage AWS resources.**\ +3. [Create an IAM user account to create and manage AWS resources](#create-an-iam-user-account-to-create-and-manage-aws-resources).\ IAM user accounts are attached to a **User Group** that has a set of specific permissions (such as reading, writing and deleting) on specified resources. We will create a User Group with predefined permissions and an IAM user account in that group. -4. **Create an alias for your account id.**\ - Your Root user account id is a 12-digit number that is difficult to remember. We are going to create an alias that is easier to remember. This is especially useful because the alias will replace the 12-digit number in the web address for logging in to your account as IAM user. +4. [Create an alias for your IAM user account](#create-an-alias-for-your-iam-user-acount).\ + Your IAM user account id is a 12-digit number that is difficult to remember. We are going to create an alias that is easier to remember. This is especially useful because the alias will replace the 12-digit number in the web address for logging in to your account as IAM user. -5. **Grant your IAM user account the permissions to access the Billing Dashboard.**\ +5. [Grant your IAM user account the permissions to access the Billing Dashboard](#grant-your-iam-user-account-the-permissions-to-access-the-billing-dashboard).\ The Billing Dashboard of your account is only accessible to the Root user by default. As you will mostly be using your IAM user account, it is convenient that you can check your bills and related information with your IAM user account too. We are going set on the permissions that enable your IAM user account to access the Billing Dashboard. ## 1. Change the default region of your account to Ireland @@ -83,7 +85,16 @@ To set up your mobile as MFA device you will need a Virtual MFA app on your mobi **If you do not have a Virtual MFA app on your mobile phone:** Go to the app store on your phone, search for Duo Mobile *or* Google Authenticator, and install it. -Once you have installed an MFA app in your mobile: - Return to the above pop-up "Set up a virtual MFA device" on your computer and choose **Show QR code**. - Open the MFA app on your phone - Press **+ Add** in Duo Mobile or **+** in Google Authenticator - Press **Use QR code** in Duo Mobile or **Scan a QR code** in Google Authenticator. Your camera will open to scan a QR code. - Point your camera at your computer screen showing the QR code to scan. You may need to adjust the zoom for the scan to occur. - Once the scan is successful, the MFA app will display a number for about 30 seconds, and then another number for the same time, and so on until you close the app. - Of those numbers shown in your mobile MFA, you need to enter two consecutive numbers into the fields **MFA Code 1** and **MFA Code 2** on the last pop-up window "Set up a virtual MFA device" on your computer. You may need to scroll down to see MFA Code 2. **NB: enter the numbers with no space between them** even if they are shown with a space in your mobile. - Click on **Assign MFA**. +Once you have installed an MFA app in your mobile: + +- Return to the above pop-up "Set up a virtual MFA device" on your computer and choose **Show QR code**. +- Open the MFA app on your phone + - Press **+ Add** in Duo Mobile or **+** in Google Authenticator + - Press **Use QR code** in Duo Mobile or **Scan a QR code** in Google Authenticator. Your camera will open to scan a QR code. + - Point your camera at your computer screen showing the QR code to scan. You may need to adjust the zoom for the scan to occur. +- Once the scan is successful, the MFA app will display a number for about 30 seconds, and then another number for the same time, and so on until you close the app. +- Of those numbers shown in your mobile MFA, you need to enter two consecutive numbers into the fields **MFA Code 1** and **MFA Code 2** on the last pop-up window "Set up a virtual MFA device" on your computer. You may need to scroll down to see MFA Code 2. **NB: enter the numbers with no space between them** even if they are shown with a space in your mobile. +- Click on **Assign MFA**. You will see a success message which you can close. @@ -95,7 +106,7 @@ We are going to create an IAM user account with which you will be able to create We will create a user group called **Administrators**, then a user account called **YourName** (your actual name), and finally attach the account to the group. As this is the first IAM group and IAM account to be created, we need to do this with the Root user account, but then it will be possible to do it with the IAM account we will create because it will have Administrator privileges. -### Create the user group +### 3.1 Create the user group Go to the IAM Dashboard page by typing **iam** in the AWS search box at the top and pressing Enter. On the IAM Dashboard, click on "User groups" under "Access Management" on the left, and then on **Create group** on the right. @@ -121,7 +132,7 @@ You now have a user group called **Administrators** ![.](/images/config-acc/ca15-iam-user-group-created.jpg){width="900px" fig-alt="Screen shot of AWS Console User groups page in a browser, with the message Loading (information) circled"} -### Create your IAM user account and add it to the Administrators group +### 3.2 Create your IAM user account and add it to the Administrators group To create your IAM user account, click on **Users** in the last page displayed in the previous step, on the left in the figure above. @@ -129,7 +140,13 @@ The page titled "Users" will be displayed. Click on **Add users**. ![.](/images/config-acc/ca16-iam-users-screen.jpg){width="900px" fig-alt="Screen shot of AWS Console IAM Users page in a browser, with the option Add users circled"} -The page below will be displayed, where you can enter your IAM user account details by: - typing your user name (**a single word** of your choice) - checking the box "Access Key - Programmatic access" --- this option enables you to use the AWS CLI - checking the box "Password - AWS Management Console access" - checking the box "Autogenerated password" and - checking the box "User must create a new password at next sign-in" +The page below will be displayed, where you can enter your IAM user account details by: + +- typing your user name (**a single word** of your choice --- we chose **adminuser**) +- checking the box "Access Key - Programmatic access" --- this option enables you to use the AWS CLI +- checking the box "Password - AWS Management Console access" +- checking the box "Autogenerated password" and +- checking the box "User must create a new password at next sign-in" Then click on **Next: Permissions** @@ -143,8 +160,8 @@ Check the box next to the group Administrators and then click on **Next: Tags**. You will be presented with a page that says "Add user - Add tags (optional)", not shown here as we are not adding tags. Click on the button **Next: Review**. -::: callout-note -Note on tags +::: callout-note +## Note on tags Adding tags --- or keywords --- to an AWS resource is optional. You don't need to tag your IAM user account because you only have one such account. Adding tags is useful when you are managing multiple user accounts/resources as it helps searching for specific resources based on their tags. ::: @@ -153,39 +170,42 @@ Your will now be presented with a page displaying the options chosen for your IA ![.](/images/config-acc/ca18-iam-user-review.jpg){width="900px" fig-alt="Screen shot of AWS Console IAM Add user Review page in a browser with the button Create user circled"} -You will now see a page with the message ***Success --- You successfully created the users shown below...*** +You will now see a page with the message ***Success*** --- *You successfully created the users shown below...* **You need to download** the .csv file indicated in this page by clicking on **Download .csv**. This file contains the **credentials** both to login to the AWS Console and to access AWS resources programmatically with your new IAM user account. *Programmatically* means access from software applications including the AWS CLI. For security reasons **you will not be able to access these credentials once you leave this page** but you can create new credentials. -Click on **Download .csv** to download and save the file in your computer. +Click on **Download.csv** to download and save the file in your computer. ![.](/images/config-acc/ca19-iam-user-created.jpg){width="900px" fig-alt="Screen shot of AWS Console IAM Add user success page in a browser with the web address to log in and button to download a .csv file circled"} ::: callout-note -## What's in the file? +## What's in the .csv file? The file you downloaded is a *comma separated value* (CSV) file that you can open in any text editor. Its content is something like this: -`User name,Password,Access key ID,Secret access key,Console login link` `adminuser,0ji)8[bN3{F-X!h,BMZ4AD..KIAVQN34,o0/bSO3WJeO..Vgtc4E3LxXZVbQg,https://xxxxxxxxxxxx.signin.aws.amazon.com/console` + +`User name,Password,Access key ID,Secret access key,Console login link` + +`adminuser,0ji)..{F-X!h,BMZ4D..QN34,o0/bSO3W..VgtXZVbQg,https://xxx..xxx.signin.aws.amazon.com/console` The first line specifies the names of the comma-separated values in the second line --- comma characters are not part of any of the values. The values in the second line shown above will be different to those in your CSV file. -The first and second fields, `adminuser` and `0ji)8[bN3{F-X!h` are the username and the password to access the AWS Console. The third and the fourth fields, `BMZ4AD..KIAVQN34` and `o0/bSO3WJeO..Vgtc4E3LxXZVbQg`, are the *access key ID* and the *secret access key* which, combined, will enable you to use the AWS CLI and, more generally, to access AWS resources programmatically. The last field, `https://xxxxxxxxxxxx.signin.aws.amazon.com/console`, is the web address to login to the AWS Console with the IAM user account you have created, and other IAM accounts you may create later. +The first and second fields, `adminuser` and `0ji)..F-X!h` are the username and the password to access the AWS Console. The third and the fourth fields, `BMZ4D..QN34` and `o0/bSO3W..VgtXZVbQg`, are the *access key ID* and the *secret access key* which, combined, will enable you to use the AWS CLI and, more generally, to access AWS resources programmatically. The last field, `https://xxx..xxx.signin.aws.amazon.com/console`, is the web address to login to the AWS Console with the IAM user account you have created. **NB**: the first time you login to the AWS Console you will have to change the password. -**NB**: we are representing here with "xxxxxxxxxxxx" the digits in the URL to login to the AWS Console, `https://xxxxxxxxxxxx.signin.aws.amazon.com/console`. This 12-digit number corresponds to your account id. +**NB**: we are using "xxx..xxx" to represent the 12 digits in the URL to login to the AWS Console, `https://xxx..xxxx.signin.aws.amazon.com/console`. The 12-digit number corresponds to your account id. ::: Once you close the success message above, in the page that appears you should see the user account you have just created, listed along with the Groups (Administrators) of which it is a member and other information, for example: "Never" under "Last activity" means you have not yet logged in. ## 4. Create an alias for your IAM user acount -A 12-digit number can be difficult to remember so let's create an alias which is easier to remember. The alias can be used to login to your account. +A 12-digit number can be difficult to remember, so let's create an alias which is easier to remember. The alias can be used to login to your account. Type **iam** in the AWS search box and press Enter to go to the "IAM Dashboard". @@ -202,7 +222,7 @@ You can now login to your account using either web address: the one with your 12 ::: callout-note ## Access your IAM user account with both URLs: -- open two **new** tabs in your browser (but do not close this browser tab so that we can finish up setting up your IAM account with the last step below). +- open two **new** tabs in your browser (but do not close this browser tab so that we can finish setting up your IAM account with the last step below). - enter `https://xxxxxxxxxxxx.signin.aws.amazon.com/console` in one of the tabs, but change "xxx..xxx" with your 12-digit account number. - enter `https://youralias.signin.aws.amazon.com/console` in the other tab, but change "youralias" with your actual account alias. - in both tabs use your actual **username** and **password** from your .csv file. diff --git a/docs/lesson01-setting-work-envs/03-configure-terminal.qmd b/docs/lesson01-setting-work-envs/03-configure-terminal.qmd index 92acda9..3b1c041 100644 --- a/docs/lesson01-setting-work-envs/03-configure-terminal.qmd +++ b/docs/lesson01-setting-work-envs/03-configure-terminal.qmd @@ -5,25 +5,26 @@ title: "Configure Your Terminal Environment" ::: callout-warning ## Prerequisites -**Please read [Workshops Organisation](/index.qmd#course-overview)** if you haven't done so. To complete this episode you will need: - -- if you are self-studying the course **or** attending a workshop using **your AWS account**: - - to have created and configured your AWS account as described in the two previous episodes: [Create Your AWS Account](./01-create-aws-account.qmd) and [Configure Your AWS Account](./02-configure-account.qmd). - - your AWS account programmatic access credentials (\*): - - Access Key ID - - Secret Access Key - - **Windows users**: to have installed Git Bash --- see the [Setup](/docs/miscellanea/precourse-instructions.qmd) section. - - **Mac users**: to have installed or updated Bash --- see the [Setup](/docs/miscellanea/precourse-instructions.qmd) section. - - **Mac** and **Linux users**: to have installed: `git`, `curl`, `unzip`, `ssh` -- if you are attending a workshop using a **Cloud-SPAN AWS account** (and an AWS Linux instance): - - to follow the instructions in this episode for Linux terminals. - - **Windows users**: to have installed Git Bash --- see [Workshops Organisation](/index.qmd#course-overview) and the [Setup](/docs/miscellanea/precourse-instructions.qmd) section. - -(\*) Those credentials are in the .csv file you downloaded once you created your IAM account as part of configuring your AWS account. +**Please read [Workshops Organisation](/index.qmd#workshops-organisation)** if you haven't done so. + +If you are attending a workshop using a **Cloud-SPAN AWS account**, you **don't need** to complete this episode. + +If you are self-studying the course **or** attending a workshop using **your AWS account**, you must complete **either** this episode or the following one: [Configure Your AWS CloudShell Environment](./04-configure-cloudshell.qmd). + +To complete this episode you will need: + +- to have created and configured your AWS account as described in the two previous episodes: [Create Your AWS Account](./01-create-aws-account.qmd) and [Configure Your AWS Account](./02-configure-account.qmd). +- your AWS account programmatic access credentials (*): + - Access Key ID + - Secret Access Key +- **Windows users**: to have installed Git Bash --- see the [Precourse Instructions](/docs/miscellanea/precourse-instructions.qmd) section. +- **Mac users**: to have installed or updated Bash --- see the [Precourse Instructions](/docs/miscellanea/precourse-instructions.qmd) section. +- **Mac** and **Linux users**: to have installed: `git`, `curl`, `unzip`, `ssh` + +(*) Those credentials are in the .csv file you downloaded once you created your IAM account as part of configuring your AWS account. ::: # Introduction - This episode will guide you to configure your **terminal environment** so that you can run the Scripts to create and manage instances. **Configuring** your terminal environment consists of: @@ -33,16 +34,17 @@ This episode will guide you to configure your **terminal environment** so that y **Installing** the Scripts and the AWS CLI includes **downloading** each and **configuring** the execution path of your terminal so that the Scripts and the AWS CLI can be run from any directory location only specifying their name. -These are the main steps you will follow to configure your Linux machine: +# Overview +These are the main steps you will follow to configure your terminal environment: -1. **Installing the Scripts**\ - You will download the Scripts from GitHub and make them accessible through the execution path in your Linux machine. +1. **[Installing the Scripts](#installing-the-scripts).**\ + You will download the Scripts from GitHub and make them accessible through the execution path of your terminal environment. -2. **Install the AWS CLI**\ - You will download the AWS CLI and make it accessible through the execution path in your Linux machine. +2. **[Install the AWS CLI](#install-the-aws-cli).**\ + You will download the AWS CLI and make it accessible through the execution path of your terminal environment. -3. **Configure the AWS CLI**\ - You will configure the AWS CLI to use the access key ID and a secret access key of your AWS IAM user account. +3. **[Configure the AWS CLI](#configure-the-aws-cli).**\ + You will configure the AWS CLI to use the access key ID and the secret access key of your AWS IAM user account. # 1. Installing the Scripts @@ -56,20 +58,27 @@ git clone https://github.com/Cloud-SPAN/aws-instances.git ~/_tmp_cloudspan_aws ## Make the Scripts accessible through the execution path -There are many ways to make the Scripts accessible through the execution path. To avoid any conflicts with the current configuration of your Linux machine, we are going to: +There are many ways to make the Scripts accessible through the execution path. To avoid any conflicts with the current configuration of your terminal environment, we are going to: 1. create a new "bin" directory 2. copy the Scripts to the new bin directory 3. add the new bin directory to the execution path -You can copy-paste the commands below to carry out those three steps using `~/.local/bincsaws` as the new bin directory, **but note**: +You can copy-paste the commands below to carry out those three steps using `~/.local/bincsaws` as the new bin directory, **but note**: **If** `~/.local/bincsaws` **already exists** in your environment (which you can check with the command `ls ~/.local`), choose another name for `bincsaws` and use the new name instead of `bincsaws` in the three commands below: -Click the bar that corresponds to your terminal to display the next command you will type or copy-paste into your terminal: +``` {.bash filename="Code"} +mkdir -p ~/.local/bincsaws +``` +``` {.bash filename="Code"} +cp ~/_tmp_cloudspan_aws/*.sh ~/.local/bincsaws +``` + +Click the bar below that corresponds to your terminal to display the next command you will type or copy-paste into your terminal: ::: {.callout-note collapse="true"} -## Git Bash and Linux terminal +## Git Bash (Windows) and Linux terminal ``` {.bash filename="Code"} echo "PATH=\"\$HOME/.local/bincsaws:\$PATH\"" >> ~/.bashrc @@ -96,19 +105,18 @@ The last command `echo "PATH=\"\$HOME ...` **adds** an assignment shell **statem The statement adds the new bin directory to the execution path which is held by the shell variable PATH. -Since the **terminal** runs the commands in that configuration file every time it is launched, that is, every time your login to your instance, the assigment shell statement (we added) will add the "new" bin directory to the execution path on every subsequent launch of the terminal, thus making the Scripts accessible from any directory location. +Since the **terminal** runs the commands in that configuration file every time it is launched, that is, every time you login to your instance, the assigment shell statement (we added) will add the "new" bin directory to the execution path on every subsequent launch of the terminal, thus making the Scripts accessible from any directory location. You **need to open** (launch) a new terminal for the execution path to get updated (you can close the old terminal as you won't use it anymore). Once you have opened a new terminal, the Scripts will be accessible through the execution path and you should be able to run the command `csinstances_create.sh` as shown below. This script is one of the Scripts installed in `~/.local/bincsaws`. ``` {.bash filename="Code"} -$ csinstances_create.sh +csinstances_create.sh ``` - The output of `csinstances_create.sh` in your terminal should look like this: -![.](/images/config-linux-env/08-linux-terminal-running-csinstances_create.png){width="900px" fig-alt="Screenshot of Linux terminal with the name of the script csinstances_create.sh circled"} +![.](/images/config-linux-env/08-linux-terminal-running-csinstances_create-ScriptsV2.png){width="900px" fig-alt="Screenshot of Linux terminal with the name of the script csinstances_create.sh circled"} The script `csinstances_create.sh` was found and run, but as it requires a parameter (the name of a file), it only displayed the usage message and finished. @@ -133,9 +141,11 @@ To install the AWS CLI in your Linux computer: The script `aws_cli_install_update_linux.sh` is one of the Scripts that you installed in the previous section. It installs or updates the AWS CLI in the directory `~/.local/bincsaws`. Open a terminal and enter its name and press `y` when prompted as to whether to continue, as shown below: -``` {.bash filename="Code"} -aws_cli_install_update.sh +```bash {filename="Code"} +$ aws_cli_install_update.sh +``` +```default {filename="Output"} aws_cli_install_update.sh installs or updates the AWS CLI and the AWS completer locally. Do you want to continue (y/n)?: @@ -169,9 +179,11 @@ echo "complete -C $HOME/MYDIRECTORY/aws_completer aws" >> ~/.bashrc Now open a terminal and run `aws_cli_install_update_linux.sh` to install the AWS CLI. Press `y` when prompted as to whether to continue: -``` {.bash filename="Code"} +```bash {filename="Code"} aws_cli_install_update.sh +``` +```default {filename="Output"} aws_cli_install_update.sh installs or updates the AWS CLI and the AWS completer locally. Do you want to continue (y/n)?: @@ -184,7 +196,7 @@ Once the script is finished, go to section 3 to configure the AWS CLI. ::: {.callout-note collapse="true"} ## **Mac instructions** -To install the AWS CLI in your Mac computer, enter or copy-paste the `brew` command below in your terminal (the [Setup](/docs/miscellanea/precourse-instructions.qmd) section shows how to install `brew` if you need to): +To install the AWS CLI in your Mac computer, enter or copy-paste the `brew` command below in your terminal (the [Precourse Instructions](/docs/miscellanea/precourse-instructions.qmd) section shows how to install `brew` if you need to): ``` {.bash filename="Code"} brew install awscli @@ -233,7 +245,10 @@ Once the install is finished, go to section 3 to configure the AWS CLI. ::: {.callout-note collapse="true"} ## Windows -To install the AWS CLI in your Windows computer, you need to use the Windows "terminal" program called `Command Prompt` as described below. Once the installation is complete, the AWS CLI will be available to the Git Bash terminal. \### **Open Windows `Command Prompt`** Go to the Windows search box (bottom left in the screenshot below) and type the word "command" in it. A window will pop up on the left, listing **Command Prompt app** at the top. Click on **Command Prompt app**. +To install the AWS CLI in your Windows computer, you need to use the Windows "terminal" program called `Command Prompt` as described below. Once the installation is complete, the AWS CLI will be available to the Git Bash terminal. + +### **Open Windows `Command Prompt`** +Go to the Windows search box (bottom left in the screenshot below) and type the word "command" in it. A window will pop up on the left, listing **Command Prompt app** at the top. Click on **Command Prompt app**. ![.](/images/config-linux-env/09-searchin-running-windows-command-prompt.png){width="900px" fig-alt="Screenshot of Windows Desktop showing Windows search box with the word command typed in circled"} @@ -273,7 +288,7 @@ Go to section 3 to configure the AWS CLI. You can use both the `Command Prompt` # 3. Configure the AWS CLI -The installation of the AWS CLI, as instructed in the previous section, is made **local** (within your HOME directory) on Linux, and **system-wide** (for all users) on Mac and Windows. Hence, Mac and Windows users may need to open a new terminal so that the execution path gets updated with the location where the AWS CLI was installed. Otherwise you may see the message `aws: command not found` when trying to configure the AWS CLI whose name as a program is `aws`. +The installation of the AWS CLI, as instructed in the previous section, is made **local** on Linux (within your HOME directory), and **system-wide** (for all users) on Mac and Windows. Hence, Mac and Windows users may need to open a new terminal so that the execution path gets updated with the location where the AWS CLI was installed. Otherwise you may see the message `aws: command not found` when trying to configure the AWS CLI whose name as a program is `aws`. To configure the AWS CLI, run the command `aws configure` as shown below, entering the following information when prompted: @@ -297,7 +312,7 @@ Default output format [None]: json To check the configuration of the AWS CLI, run the following command: ``` {.bash filename="Code"} -$ aws ec2 get-vpn-connection-device-types +aws ec2 get-vpn-connection-device-types ``` If your configuration is correct, you should see an output like this: diff --git a/docs/lesson01-setting-work-envs/04-configure-cloudshell.qmd b/docs/lesson01-setting-work-envs/04-configure-cloudshell.qmd index cec619d..e3e0ce1 100644 --- a/docs/lesson01-setting-work-envs/04-configure-cloudshell.qmd +++ b/docs/lesson01-setting-work-envs/04-configure-cloudshell.qmd @@ -5,12 +5,17 @@ title: "Configure Your AWS CloudShell Environment" ::: callout-warning ## Prerequisites -**Please read [Workshops Organisation](/index.qmd#course-overview)** if you haven't done so. To complete this episode you will need: +**Please read [Workshops Organisation](/index.qmd#workshops-organisation)** if you haven't done so. + +If you are attending a workshop using a **Cloud-SPAN AWS account**, you **don't need** to complete this episode. + +If you are self-studying the course **or** attending a workshop using **your AWS account**, you must complete **either** this episode or the previous one: [Configure Your Terminal Environment](./03-configure-terminal.qmd). + +To complete this episode you will need: + +- to have created and configured your AWS account as described in the episodes [Create Your AWS Account](./01-create-aws-account.qmd) and [Configure Your AWS Account](./02-configure-account.qmd). +- the **login page**, **username** and **password** of your IAM user account. You can find them in the .csv file you downloaded once you created your IAM account as part of configuring your AWS account. -- if you are self-studying the course **or** attending a workshop using **your AWS account**: - - to have created and configured your AWS account as described in the two previous episodes: [Create Your AWS Account](./01-create-aws-account.qmd) and [Configure Your AWS Account](./02-configure-account.qmd). - - the **login page**, **username** and **password** of your IAM user account. -- if you are attending a workshop using a **Cloud-SPAN AWS account** (and an AWS Linux instance), you **don't need** to complete this episode but to **follow the instructions** in the episode [Configure Your Terminal Environment](./03-configure-terminal.qmd). ::: # Introduction @@ -25,17 +30,19 @@ The AWS CloudShell: - has the **AWS CLI** installed and configured with **programmatic access** to your AWS account using temporary security credentials based on the credentials to access the AWS Console --- this is completely transparent to the user. - has other software tools installed including: `Python`, `Node.js`, `git`, `make`, `pip`, `curl`, `zip`, `unzip`, `ssh`. - runs on a Linux AWS instance whose input and output are directed to the browser tab where it is invoked. The instance has 1 GB storage as a Linux file system. -- preserves data and software that you install in your home directory between sessions, but only **for 120 days** --- you will receive warning emails before they are deleted so you can access the AWS CloudShell again. You can install software at **system-wide level** but you will have to reinstall it every time you access the AWS CloudShell. +- preserves data and software that you install within your **home directory** between sessions, but only **for 120 days** --- you will receive warning emails before they are deleted so you can access the AWS CloudShell again. You can install software at **system-wide level** but you will have to reinstall it every time you access the AWS CloudShell. For some applications, the key bindings in the AWS CloudShell may be slightly different to the key bindings for same applications in a desktop/laptop machine. +# Overview + These are the main steps you will follow to configure your AWS CloudShell environment: -1. **Download the Scripts from GitHub**\ +1. [Download the Scripts from GitHub](#download-the-scripts-from-github).\ You will login to the AWS Console with your IAM user account, launch the AWS CloudShell, and download the Scripts from GitHub. -2. **Make the Scripts accessible through the execution path**\ - You will configure the AWS CloudShell so that the Scripts can be run from any location in the AWS CloudShell file system. +2. [Make the Scripts accessible through the execution path](#make-the-scripts-accessible-through-the-execution-path).\ + You will configure your AWS CloudShell environment so that the Scripts can be run from any location in the AWS CloudShell file system. # 1. Download the Scripts from GitHub @@ -58,11 +65,11 @@ An AWS CloudShell that you launch can only operate on resources within a selecte If you are using an **institutional account** and you cannot set the region to Ireland, set it to the region which your institutional account is enabled to. Hence you will be installing and running the Scripts in the CloudShell for that region. However, before running the Scripts (in the next lesson), you will need to ask your IT department to copy the Cloud-SPAN AMI to your AWS account. -(Note: you can launch multiple AWS CloudShells, each operating on a different region, but this is not needed and hence not covered in the course.) +**Note**: you can launch multiple AWS CloudShells, each operating on a different region, but this is not needed and hence not covered in the course. ## Launch the AWS CloudShell -To launch the AWS CloudShell, click on its icon at the top: the little square with these two characters "\*\*\>\_\*\*" inside. +To launch the AWS CloudShell, click on its icon at the top: the little square with these two characters "**\>\_**" inside. ![.](/images/config-linux-env/01-console-login-as-iam-user.png){width="900px" fig-alt="Screenshot of AWS Console page in a browser with the region menu at the top right and AWS CloudShell option at middle top circled"} @@ -76,19 +83,17 @@ Click on **Close** to get to the AWS CloudShell. You will now be presented with the AWS CloudShell terminal, like the one below. -It may take up to a couple of minutes for the AWS CloudShell terminal to be ready the first time you open it in a Console session: - -- first, an AWS instance called **the environment** will be launched --- you will see the message "Waiting for the environment to run ...", not shown below -- then the **terminal** program will be run --- you will see the message "Preparing your terminal ...". -- finally, you will see the terminal **prompt**, something like this: [**cloudshell-user @ip-11-22-33-44 \~**]**\$** --- the numbers in the prompt are the IP address of the instance and may differ every time you open the AWS CloudShell. +It may take up to a couple of minutes for the AWS CloudShell terminal to be ready the first time you open it in a Console session because: -The message after the first prompt, "Try these commands to get started: aws help ...", means that the AWS CLI, whose name when used is `aws`, is ready to be used. +- first, an AWS instance called **the environment** will be launched --- you may see one of these messages (not shown below) "Waiting for the environment to run .." or "Creating the environment ..". +- then the **terminal** program will be run --- you may see the message "Preparing your terminal ...". +- finally, you will see the terminal **prompt**, something like this: [**cloudshell-user @ip-10-134-43-226 \~**]**\$** --- the numbers in the prompt are the IP address of the instance and may differ every time you open the AWS CloudShell t. ![.](/images/config-linux-env/04-aws-cloudshell-ready-screen.png){width="900px" fig-alt="Screenshot of AWS Console page in a browser showing the AWS CloudShell terminal prompt"} ## Download the Scripts -To download the Scripts, enter or copy-paste the git command below into the terminal and press Enter. +To download the Scripts, enter or copy-paste the git command below into the CloudShell terminal and press Enter. ``` {.bash filename="Code"} git clone https://github.com/Cloud-SPAN/aws-instances.git ~/_tmp_cloudspan_aws @@ -99,14 +104,13 @@ Now enter the command `ls -a ~` to list all the files (visible and hidden) in yo ``` {.bash filename="Code"} ls -a ~ ``` - -The terminal should now look similar to the following. If so, you have downloaded the Scripts into the \*\*\_tmp_cloudspan_aws\*\* directory in your home directory. +The CloudShell terminal should now look similar to the following. If so, you have downloaded the Scripts into the **\_tmp_cloudspan_aws** directory in your home directory. ![.](/images/config-linux-env/05-aws-cloudshell-cloning-the-scripts.png){width="900px" fig-alt="Screenshot of AWS Console page in a browser showing the AWS CloudShell terminal with git command line, the ls command line and download directory circled"} # 2. Make the Scripts accessible through the execution path -There are many ways to make the Scripts accessible through the execution path. To avoid any conflicts with the current configuration of the AWS CloudShell, we are going to: +There are many ways to make the Scripts accessible through the execution path. To avoid any conflicts with the current configuration of the CloudShell terminal, we are going to: 1. create a new "bin" directory 2. copy the Scripts to the new bin directory @@ -124,13 +128,13 @@ echo "PATH=\"\$HOME/.local/bincsaws:\$PATH\"" >> ~/.bashrc The command `echo ... >> ~/.bashrc` adds an assignment shell statement at the end of the Bash configuration file `~/.bashrc`. The statement adds the new bin directory to the execution path which is held by the shell variable `PATH`. -Since the *terminal* program runs the commands in `~/.bashrc` every time it is launched, the assigment shell statement will add the "new" bin directory to the execution path on every subsequent launch of the AWS CloudShell. +Since the *terminal* program runs the commands in `~/.bashrc` every time it is launched, the assigment shell statement will add the "new" bin directory to the execution path on every subsequent launch of the CloudShell terminal. That means that the Scritps **are not yet accessible through the execution path**. -For the Script to become thus accessible, we need either to restart the *terminal*, or to restart the AWS CloudShell, or to run the `source` command to make the **current** *terminal* run the commands in `~/.bashrc` again, and thus get the PATH variable updated within this terminal session. +For the Script to become thus accessible, we need either to restart the CloudShell *terminal*, or to restart the CloudShell environment (the AWS instance mentioned earlier), or to run the `source` command to make the **current** CloudShell *terminal* run the commands in `~/.bashrc` again, and thus get the PATH variable updated within this terminal session. -Let's use the `source` command to test the Scripts are accessible through the execution path, and then look at how to restart the terminal and the AWS CloudShell. +Let's use the `source` command to test the Scripts are accessible through the execution path, and then look at how to restart the CloudShell terminal and the CloudShell environment. Enter or copy-paste the following commands: @@ -140,7 +144,7 @@ source ~/.bashrc csinstances_create.sh ``` -The terminal should now look like this: +The CloudShell terminal should now look like this: ![.](/images/config-linux-env/06-aws-cloudshell-installing-the-scripts-n-running-csinstances_create.png){width="900px" fig-alt="Screenshot of AWS Console page in a browser showing the AWS CloudShell terminal with command lines for installing and running one of the Scripts circled"} @@ -156,12 +160,12 @@ Don't delete the directory `~/_tmp_cloudspan_aws` where the Scripts were downloa Please don't delete that directory just yet. We will use some files there in the next lesson. Once we use those files you will delete that directory. ::: -### Restarting the terminal and the AWS CloudShell +### Restarting the CloudShell terminal and the CloudShell environment You can restart the terminal by simply logging out of the terminal through entering the command `exit` or typing `Ctrl-d` (pressing the keys `Ctrl` and `d` simultaneously). Once you are logged out, press any key for the *terminal* to be launched again. Restarting the terminal is lightweight in that it is relatively quick because only the terminal program is restarted. -You can restart the AWS CloudShell by closing the browser tab and launching CloudShell again, or by clicking on the **Actions** drop-down menu on the top right of the screen and then clicking on **Restart AWS CloudShell**. +You can restart the CloudShell environment by closing the browser tab and launching CloudShell again, or by clicking on the **Actions** drop-down menu on the top right of the screen and then clicking on **Restart**. -![.](/images/config-linux-env/07-aws-cloudshell-actions-menu.png){width="900px" fig-alt="Screenshot of AWS Console page in a browser showing the AWS CloudShell terminal with the options Actions and Restart AWS Cloudshell circled"} +![.](/images/config-linux-env/07-aws-cloudshell-actions-menu.png){width="900px" fig-alt="Screenshot of AWS Console page in a browser showing the AWS CloudShell terminal with the options Actions and Restart circled"} Restarting the AWS CloudShell through that menu is heavyweight in that it will first stop and then relaunch the AWS Linux instance on which the AWS CloudShell terminal runs. It may be useful if you get stuck within a program and you don't know how to get out (it happened to one of the authors). diff --git a/docs/lesson01-setting-work-envs/index.qmd b/docs/lesson01-setting-work-envs/index.qmd index 876dd1b..557c1d7 100644 --- a/docs/lesson01-setting-work-envs/index.qmd +++ b/docs/lesson01-setting-work-envs/index.qmd @@ -2,30 +2,33 @@ title: Setting Up Your Cloud and Terminal Environments --- -Please read [**Workshops Organisation**](/index.qmd#workshops-organisation) if you haven't done so. +**Please read [Workshops Organisation](/index.qmd#workshops-organisation)** if you haven't done so. The Bash shell scripts that automatically manage multiple AWS instances will be referred to as the "Scripts" from now on. You can use any of the following terminals to run the Scripts as instructed in this course: - Linux terminals that run the Bash shell. -- Windows Git Bash terminals --- see the [Setup](/docs/miscellanea/precourse-instructions.qmd) section. -- Mac terminals that run the Bash shell or the Zsh shell --- see the [Setup](/docs/miscellanea/precourse-instructions.qmd) section to install or update Bash; Bash version must be 5.0 or higher. +- Windows Git Bash terminals --- see the [Precourse Instructions](/docs/miscellanea/precourse-instructions.qmd) section to install Git Bash. +- Mac terminals that run the Bash shell or the Zsh shell --- see the [Precourse Instructions](/docs/miscellanea/precourse-instructions.qmd) section to install or update Bash; Bash version must be 5.0 or higher. - the **AWS CloudShell** terminal, a Linux terminal that runs the Bash shell by default. The AWS CloudShell is hosted on AWS and is used through the browser. # Overview The Scripts make use of the **AWS Command Line Interface** (AWS CLI), a software tool that enables you to interact with AWS through **commands** that can be run either within **shell scripts** or in any of those **terminals** above (and others). The Scripts run the AWS CLI to make requests to manage (create, allocate, ..., and delete) AWS services such as instances, storage, domain names, etc. For such requests to be successful, the **AWS CLI** and the target **AWS account**, wherein services will be managed, **need to be configured**. -This lesson will guide you to: - create your AWS account. - configure your AWS account for daily work, including **enabling access with the AWS CLI**. - configure your **terminal environment** with the Scripts and the AWS CLI, the AWS CLI configured to manage resources within your AWS account. +This lesson will guide you to: -### Your AWS account +- create your AWS account --- Episode 1 (see menu on the left). +- configure your AWS account for daily work, including **enabling access with the AWS CLI** --- Episode 2. +- configure your **terminal** environment with the Scripts and the AWS CLI configured to manage resources within your AWS account --- Episode 3. +- configure your **AWS CloudShell** terminal environment with the Scripts (the AWS CLI is already configured to manage resources within your AWS account) --- Episode 4. -Episodes 1 and 2 provide the instructions to create and configure your AWS account. +### Your AWS account -We organised the instructions in all episodes (4 episodes in lesson 1 and 4 in lesson 2) assuming that you are going to **create and use your own** AWS account, and hence you have full permissions to configure it as instructed in the course --- we refer to such an account as an AWS **personal account**. +We organised the instructions in this course assuming that **you are going to create and use your own** AWS account, and hence you have full permissions to configure it as instructed in the course --- we refer to such an account as an AWS **personal account**. -However, **you can use an existing AWS account** you have access to. Throughout the course, where relevant, we point out what "extra steps" you may need to do to configure your account as required to run the Scripts. +However, **you can use an existing AWS account** you already have access to. Throughout the course, where relevant, we point out what "extra steps" you may need to do to configure your account as required to run the Scripts. If you already have an AWS account that you would like to use to run the Scripts, skip Episode 1 and configure your account as described in Episode 2. Specifically, to enable **access with the AWS CLI**, you need to have/create an IAM (Identity and Access Management) user account enabled with **programmatic access** based on **Access key ID** and **Secret access key** credentials. If the AWS account you are using is your personal account, you have full permissions to do the configuration presented in that section. @@ -48,7 +51,7 @@ The course does not cover the configuration and use of those or similar security Episode 3 provides the instructions to install the Scripts and the AWS CLI in your Git Bash, Linux or Mac terminal environment, and to configure the AWS CLI to use your credentials (private and public keys) when invoked within the Scripts to request AWS service operations. The credentials will be generated using the AWS Console when you configure your AWS account in Episode 2. -Episode 4 provides the instructions to configure the Scripts in the AWS CloudShell. The AWS CloudShell is "a browser-based \[Linux Bash terminal\] shell that gives you command-line access to your AWS resources" and **has the AWS CLI** already **installed and ready to run** using "credentials" derived from the information you use to login to the AWS Console. Hence you only need to configure the Scripts, but **you must be logged in** to your AWS account through **the AWS Console** to be able **to run the AWS CloudShell** and the Scripts. +Episode 4 provides the instructions to configure the Scripts in the AWS CloudShell. The AWS CloudShell is "a browser-based \[Linux Bash terminal\] shell that gives you command-line access to your AWS resources", and **has the AWS CLI** already **installed and ready to run** using "credentials" derived from the information you use to login to the AWS Console. Hence you only need to configure the Scripts. **You must be logged in** to your AWS account through **the AWS Console** to be able **to run the AWS CloudShell** and the Scripts. In this course you will be using the AWS Console and, through the Scripts, the AWS CLI to manage AWS services but there are other ways to manage AWS services, see the callout below. diff --git a/docs/lesson02-managing-aws-instances/01-configure-instances-internet.qmd b/docs/lesson02-managing-aws-instances/01-configure-instances-internet.qmd index 8dcfbb9..b1711e1 100644 --- a/docs/lesson02-managing-aws-instances/01-configure-instances-internet.qmd +++ b/docs/lesson02-managing-aws-instances/01-configure-instances-internet.qmd @@ -3,34 +3,39 @@ title: "Configure Instances Internet Access" --- ::: callout-warning ## Prerequisites -**Please read [Workshops Organisation](https://cloud-span.github.io/cloud-admin-guide-0-overview#course-overview)** if you haven't done so. To complete this episode you will need: +**Please read [Workshops Organisation](/index.qmd#workshops-organisation)** if you haven't done so. -- if you are self-studying the course **or** attending a workshop using **your AWS account**: - - to have created and configured your AWS account as described in Episodes 1 and 2 of Lesson 1, namely: [Create Your AWS Account](/docs/lesson01-setting-work-envs/01-create-aws-account.qmd) and [Configure Your AWS Account](/docs/lesson01-setting-work-envs/02-configure-account.qmd). - - the AWS Console login details of your IAM user account: **login page**, **username** and **password**. -- if you are attending a workshop using a **Cloud-SPAN AWS account** (and an AWS Linux instance), you **don't need** to complete this episode. -::: +If you are attending a workshop using a **Cloud-SPAN AWS account**, you **don't need** to complete this episode. + +If you are self-studying the course **or** attending a workshop using **your AWS account**, you must complete this episode and for that you will need: + +- to have created and configured your AWS account as described in the episodes [Create Your AWS Account](/docs/lesson01-setting-work-envs/01-create-aws-account.qmd) and [Configure Your AWS Account](/docs/lesson01-setting-work-envs/02-configure-account.qmd). +- the AWS Console login details of your IAM user account: **login page**, **username** and **password**. +::: -# Outline -## Steps +# Overview These are the main steps you will follow to configure internet access for the instances you will create with the Scripts: -1. **Create Your Base Domain Name**\ -The base domain name you will create will be configured so that the Scripts can create a sub-domain name for each instance. If you are using an **AWS institutional account**, your base domain name may itself be a sub-domain within a domain name managed by your institution. Check the callout at the end of this section to ensure your based domain name is properly configured by you or your IT department. +1. **[Create Your Base Domain Name](#create-your-base-domain-name)**.\ +**This step is optional**. If you decide not to create a base domain name, or do not specify it (as described in the next episode) when creating instances, the instances created by the Scripts will be accessed (logged in) using the IP addresses allocated to the instancs by AWS.\ +If you decide to create a base domain name, and specify it (as described in the next episode) when creating instances, the Scripts will create a sub-domain to access each instance. If you are using an **AWS institutional account**, your base domain name may itself be a sub-domain within a domain name managed by your institution. Check the callout at the end of this section to ensure your based domain name is properly configured by you or your IT department. -2. **Create Your AWS Security Group**\ +2. **[Create Your AWS Security Group](#create-your-aws-security-group)**.\ An AWS **security group** is a set of rules to configure the Internet communication ports of AWS instances. The security group you will create will enable the communication port used by the `ssh` program. We only need one security group as all instances need the same configuration. If you are using an **AWS institutional account**, you should not have any problem creating a security group --- try to create it; if you cannot then contact your IT deparment. -3. **Select an AWS Subnet to Access the Instances**\ -An AWS subnet is a network within an AWS region to which the instances created by the Scripts will be connected so that they can be accessed from anywhere. You will select one of the subnets available in the Ireland region. If you are using an **AWS institutional account**, you may be able or need to use a subnet already setup by your IT department --- ask them. +3. **[Select an AWS Subnet to Access the Instances](#select-an-aws-subnet-to-access-the-instances)**.\ +An AWS subnet is a network within an AWS region to which the instances created by the Scripts will be connected so that they can be accessed from anywhere.\ +**This step is optional**. If you do not specify a **subnet-id** (as described in the next episode) when creating instances, the Scripts will try to obtain a subnet-id from your AWS account. We have successfully tested the Scripts to obtain and use a subnet-id running the scripts with a personal AWS account and with an institutional AWS account. The personal account was configured to use the default subnets allocated by AWS. The institutional account was configured **not** to use the default subnets allocated by AWS, but other subnets configured by the IT department of our institution.\ +If the scripts cannot obtain a subnet-id, they will display a message describing how to obtain a subnet-id (see below the detailed instructions for this step): to search for vpc (virtual private cloud) in the AWS Console (in the search box at the top), then click on subnets on the left menu, and to copy one of the subnet-id's displayed. **NB**: if no subnet-id is displayed in the AWS Console, and you are using an **AWS institutional account**, you will need to contact your IT department. + +You must do the above steps using the AWS Console with your IAM user account. These are the instructions to **login to your IAM account**: -You are going to create those three resources using the AWS Console with your IAM user account. These are the instructions to **login to your IAM account**: - Open a new browser window and enter the address of the login page for your account IAM users. The address contains your account alias or your 12-digit account number and has this form: - https://**youraccountalias**.signin.aws.amazon.com/console - https://**123456789012**.signin.aws.amazon.com/console - Enter your IAM username and password. You may also get a **Security check** to complete. -- **Set the region to Ireland** (eu-west-1) if it is not, by clicking on the drop-down menu on the top to the right, see top right on the page below. +- **Set the region to Ireland** (eu-west-1) if it is not, by clicking on the drop-down menu at the top to the right, see top right on the page below. # 1. Create Your Base Domain Name ### Go to the Route 53 Dashboard @@ -103,13 +108,13 @@ In the page that appears, "your-domain-name", click **Hosted zone details**, and ## If you are using an institutional AWS account: Recall that the base domain name `awsplaicloud.com` was created with an AWS **personal account** and that instances domain names will have the form `instancename-001.awsplaicloud.com`, where `instancename-001` will be a subdomain name within `awsplaicloudcom`. -If you are using an AWS **institutional account**, your base domain name will likely be something like `cloud-span.aws.york.ac.uk`, and instances domain names will have the form `instancename-001.cloud-span.aws.york.ac.uk`, where: +Using an AWS **institutional account**, your base domain name will likely be something like `cloud-span.aws.york.ac.uk`, and instances domain names will have the form `instancename-001.cloud-span.aws.york.ac.uk`, where: - `york.ac.uk` is the **base** domain name of the University of York, and has two parts: the domain name *per se*, `york`, and the domain suffix or top-level domain (TLD), `ac.uk`, which must be chosen among various other suffixes such as: `.com`, `.edu`, `.net`, `.org`, `.link`, etc. - `aws` is a subdomain managed by the IT department to identify AWS domains used by a department or project within the university --- similar subdomains are `gcp` for Google Cloud Platform, and `azure` for Microsoft cloud platform. - `cloud-span` is the subdomain configured by the IT department for the Cloud-SPAN project. Note that words separated by a hyphen (-) count as a single word in domain and subdomain names. It is the dot `.` that separates a domain name into subdomains, though some subdomains pairs are handled as one, for example: `co.uk` is handled as one TLD by AWS and probably by other providers of domain names. - `instancename-001` is the subdomain for the instance-001 created by the Scripts. -- **IMPORTANT**: while both `cloud-span` and `instancename-001` are both subdomains (within `aws.york.ac.uk` and `cloud-span.aws.york.ac.uk` respectively), they are configured differently: +- **IMPORTANT**: while both `cloud-span` and `instancename-001` are both subdomains (respectively within `aws.york.ac.uk` and `cloud-span.aws.york.ac.uk`), they are configured differently: - `cloud-span` is configured as a **host zone** (as `awsplaicloud.com` was configured above), and hence has an AWS **host zone id**. You need to configure, or ask your IT department to configure, your base domain name as a host zone so you can configure the Scripts with the corresponding **host zone name** and **host zone id**. - `instancename-001`, `instancename-002`.., `instancename-999` are configured by the Scripts as **A Records** (which map domain names to IPv4 addresses) within the specified host zone. ::: @@ -166,6 +171,7 @@ You should now see a page like the one below showing the details of your securit ![.](/images/01-instances-internet-access/19-ec2-security-groups-inbound-rules-created.png){width="900px" fig-alt="Screenshot of AWS Console Cloud-SPAN Security Group page in a browser showing the security group ID circled but hidden."} # 3. Select an AWS Subnet to Access the Instances + ### Go to the Virtual Private Cloud (VPC) Dashboard To select an AWS Subnet, type **vpc** in the AWS search box at the top and press Enter. Check that you are in the **Ireland** region and set it thus if not. diff --git a/docs/lesson02-managing-aws-instances/02-instances-management.qmd b/docs/lesson02-managing-aws-instances/02-instances-management.qmd index 73d884e..841b22d 100644 --- a/docs/lesson02-managing-aws-instances/02-instances-management.qmd +++ b/docs/lesson02-managing-aws-instances/02-instances-management.qmd @@ -3,44 +3,47 @@ title: "Instances Management Tasks Using the Scripts" --- ::: callout-warning ## Prerequisites -**Please read [Workshops Organisation](/index.qmd#course-overview)** if you haven't done so. To complete this episode you will need: - -- if you are self-studying the course **or** attending a workshop using **your AWS account**: - - to have created your AWS account as described in [Create Your AWS Account](/docs/lesson01-setting-work-envs/01-create-aws-account.qmd) (Episode 1, Lesson 1). - - to have configured your AWS account as described in [Configure Your AWS Account](/docs/lesson01-setting-work-envs/02-configure-account.qmd) (Episodes 2, Lesson 1). - - to have configured your terminal environment as described in either of these episodes: - - [Configure Your Terminal Environment](/docs/lesson01-setting-work-envs/03-configure-terminal.qmd) (Episode 3, Lesson 1) --- **or** - - [Configure Your AWS CloudShell Environment](/docs/lesson01-setting-work-envs/04-configure-cloudshell.qmd) (Episode 4, Lesson 1) - - to have configured instances internet access as described in [Configure Instances Internet Access](./01-configure-instances-internet.qmd) (previous episode, this lesson). - - your **base domain name**. - - the AWS resource IDs of your: **host zone**, **security group**, and **subnet**. - - the AWS Console login details of your IAM user account: **login page**, **username** and **password**. -- if you are attending a workshop using a **Cloud-SPAN AWS account** (and an AWS Linux instance), you **will be given** the necessary information at the workshop. +**Please read [Workshops Organisation](/index.qmd#workshops-organisation)** if you haven't done so. + +If you are attending a workshop using a **Cloud-SPAN AWS account**, you **will be given** the necessary information at the workshop. + +If you are self-studying the course **or** attending a workshop using **your AWS account**, you will need: + +- to have created your AWS account as described in episode [Create Your AWS Account](/docs/lesson01-setting-work-envs/01-create-aws-account.qmd). +- to have configured your AWS account as described in episode [Configure Your AWS Account](/docs/lesson01-setting-work-envs/02-configure-account.qmd). +- to have configured your terminal environment as described in either of these episodes: + - [Configure Your Terminal Environment](/docs/lesson01-setting-work-envs/03-configure-terminal.qmd) --- **or** + - [Configure Your AWS CloudShell Environment](/docs/lesson01-setting-work-envs/04-configure-cloudshell.qmd) +- to have configured instances internet access as described in episode [Configure Instances Internet Access](./01-configure-instances-internet.qmd) to get your: + - **host zone** (base domain name) and **host-zone-id** ------ ***OPTIONAL*** (read the Overview of that episode). + - **security-group-id** + - **subnet-id** ------ ***OPTIONAL*** (idem.) +- the AWS Console login details of your IAM user account: **login page**, **username** and **password**. ::: # Overview This episode will guide you through the main tasks involved in creating and managing AWS instances for a course/workshop, including: configuring the Scripts, running the Scripts, and auxiliary tasks such as managing worshop cancellations and troubleshooting. -## Sections -1. **The Scripts Running Environment**\ +#### **Sections** +1. **[The Scripts Running Environment](#the-scripts-running-environment)**\ This section introduces the Scripts, the directory-file structure the Scripts require to run successfully, and the **constraints you need to observe in configuring and running the Scripts**. -2. **Configure the Scripts Running Environment**\ +2. **[Configure the Scripts Running Environment](#configure-the-scripts-running-environment)**\ You will create a running environment for the Scripts within your terminal environment: a Git Bash terminal, a Linux terminal, a Mac terminal or the AWS CloudShell terminal. You will create the directory-file structure that will enable the Scripts to manage a few AWS instances. -3. **Create the Instances**\ +3. **[Create the Instances](#create-the-instances)**\ You will create the instances specified in Section 2 and this will generate some results. The results generated and displayed to the terminal by the Scripts that create AWS resources (instances, login key files, etc.) will be explained to help you understand how the Scripts work. -4. **Check Instances Are Accessible**\ +4. **[Check the Instances Are Accessible](#check-the-instances-are-accessible)**\ Once instances are created, it is convenient to check that they are accessible through `ssh` by logging in to a few of them. This section will show you how to login to instances in a way easier than using `ssh` directly. -5. **Understand the Results of Creating Instances**\ +5. **[Understand the Results of Creating Instances](#understand-the-results-of-creating-instances)**\ Creating AWS resources involves making AWS requests that return back results that are saved to files within the Scripts running environment. These files are crucial for the successful operation of the Scripts in stopping, starting and eventually deleting the created resources. These files are explained in this section to round up your understanding of how the Scripts work. -6. **Typical Instances Life Cycle: create, stop, start and delete instances**\ +6. **[Typical Instances Life Cycle: create, stop, start and delete instances](#typical-instances-life-cycle-create-stop-start-and-delete-instances)**\ This section describes the typical use-case scenario of instances management for a workshop /course. It describes our approach to instances management using both the **terminal** and the AWS Console. This section also describes **how to create instances in two steps** in order to reduce costs when deploying relatively large instances in terms of storage. -7. **Unforeseen Instance Management**\ +7. **[Unforeseen Instance Management](#unforseen-instance-management)**\ This section describes our approach to handle unforeseen instance management requests such as creating additional instances for a workshop due to late registrations, or deleting some (not all) instances before the end of a workshop due to cancellations. 8. **[Troubleshooting](#troubleshooting)**\ @@ -49,90 +52,80 @@ This section presents some problems we have come across in managing instances an # 1. The Scripts Running Environment These are Scripts that create and manage AWS instances -``` {.bash filename="Output"} +``` {.default filename="Listing"} aws_domainNames_create.sh aws_instances_terminate.sh csinstances_create.sh aws_domainNames_delete.sh aws_loginKeyPair_create.sh csinstances_delete.sh aws_instances_configure.sh aws_loginKeyPair_delete.sh csinstances_start.sh aws_instances_launch.sh colour_utils_functions.sh csinstances_stop.sh ``` -The four scripts `csinstances_*.sh` (on the right) are to be run by the **user of the Scripts**, the person in charge of creating, configuring, stopping, starting and deleting AWS instances for a workshop /course. The scripts `aws_*.sh` are invoked by the scripts `csinstances_create.sh` or `csinstances_delete.sh` to either create or delete instances and the corresponding domain names, IP addresses, and login keys. The file `colours_utils_functions.sh` provides (is "sourced" by) the other scripts with text colouring functions (for the results of the other scripts to be easier to read) and other utility functions. +The four scripts `csinstances_*.sh` (on the right) are to be run by the **admin user** of the Scripts, the person in charge of creating, configuring, stopping, starting and deleting AWS instances for a workshop /course. The scripts `aws_*.sh` are invoked by the scripts `csinstances_create.sh` or `csinstances_delete.sh` to either create or delete instances and related resources, that is: domain names, IP addresses, and login keys. -### Configuring the directory structure +The file `colours_utils_functions.sh` provides (is "sourced" by) the other scripts with text colouring functions (for the results of the other scripts to be easier to read) and utility functions that validate the invocation and results of the other scripts. -To run the Scripts successfully we first need to specify the following three items of information, each in a separate file: -- the names of the instances to create (delete, etc.) -- the AWS resources to be used: the Amazon Machine Image (AMI) template, base domain name, etc. -- the tags to be used to label the instances and related resources +### File-directory structure to manage instances for workshops +You have configured your AWS account and terminal environment as described in the previous episodes. Now, prior to creating instances for a workshop, you must create and organise **three files** as follows: -The three files with that information must be placed **inside** a directory called `inputs`, and the `inputs` directory must be placed **inside** another directory whose name you can choose. We use this directory structure: +- *instancesNamesFile.txt* --- contains the names of the instances to be created and managed. **Only the name of this file can be changed if preferred**. This file must contain only one instance name per line, and each instance name must start with an alphabetic character followed by alpha-numeric characters, hyphens (-) or underscores (_) only. -``` {.bash filename="Output"} -courses ### you can omit this directory or use other name - genomics01 ### course/workshop name; you can use other name - inputs ### you **cannot** use other name - instancesNames.txt ### you can use other name - resourcesIDs.txt ### you **cannot** use other name - tags.txt ### you **cannot** use other name - outputs ### created automatically by the Scripts - not to be changed - genomics02 ### another course: inputs and outputs dirs. inside - metagenomics01 ### another course: inputs and outputs dirs. inside -``` - -We handle a *courses* directory that contains a directory for each course /workshop that we run, for example: *genomics01*, *genomics02*, *metagenomics01*, etc. Each course directory has its `inputs` directory and inside the three three files mentioned above: *instancesNames.txt*, `resourcesIDs.txt`, and `tags.txt` (note that we are using *this style* for file/directory names you can choose and `this style` for names you cannot change). - -You must create the directory structure above, or a similar one, and those three files before running the Scripts to create the instances for a course. The `outputs` directory inside each course/workshop directory is created automatically by the Scripts to save the **results** of running the Scripts. - -### Configuring the file *instancesNames.txt* - -You **can use another name** for the file *instancesNames.txt*. Inside the file you must specify each of the names of the instances to create (delete, etc.) in one line, like this: - -``` {.bash filename="Output"} -instance01 -instance02 -... -instance99 -``` +- `resourcesIDs.txt`--- contains a set of space-separated “key value” pairs that specify the +AWS resources to use in creating each instance and related resources. This is the +contents of the `resourcesIDs.txt` file we use to create instances for the [Cloud-SPAN Genomics course](https://cloud-span.github.io/00genomics/): -Instances names can use any alpha-numeric characters and hyphens/minus signs (`-`), for example: *genomics-instance01*, *metagenomics-course-instance-01*. + ```{.default filename="resourcesIDs.txt for Cloud-SPAN Genomics course"} + KEYWORD VALUE examples (Cloud-SPAN's for Genomics course using domain names) + ## NB: "key value" pairs can be specified in any order + imageId ami-07172f26233528178 ## NOT optional: instance template (AMI) .. + instanceType t3.small ## NOT optional: processor count, memory .. + securityGroupId sg-0771b67fde13b3899 ## NOT optional: should allow ssh (port 22) + subnetId subnet-00ff8cd3b7407dc83 ## optional: search vpc in AWS console & .. + hostZone cloud-span.aws.york.ac.uk ## optional: specify to use domain names + hostZoneId Z012538133YPRCJ0WP3UZ ## optional: specify to use domain names + ``` -### Configuring the file `resourcesIDs.txt` -You **cannot use another name** for the file `resourcesIDs.txt`. Inside the file you must specify all the following items of information: + + -``` {.bash filename="Output"} -imageId ami-00c0ea23e53f48472 -instanceType t3.small -securityGroupId sg-0771b63b387fde199 -subnetId subnet-00ff83b7407dcdc83 -hostZone cloud-span.aws.york.ac.uk -hostZoneId Z01RCJ0WP2538133YP3UZ -``` + As shown in this example, a `resourcesIDs.txt` file can have comments in addition to the “key value” pairs to specify. The “key value” pairs can be specified in any order, but each key word must be the first item in a line and its corresponding value the second item in the same line. The key words in the example must be used, but they are NON-case sensitive. The three not optional “key value” pairs must be specified. -You **cannot change the values on the left column**. + The values are all validated. The value of `imageId` is validated to correspond to an AMI in the user’s AWS account or to a public AMI available in the AWS region on which the scripts are running. The value of `instanceType` is validated to be a valid AWS instance type. The values of `securityGroupId`, `subnetId`, `hostZone` and `hostZoneId` are validated to exist in the user’s AWS account. -You can and should change the values on the right column. For this course, however, you will use the values specified for `imageId` and `instanceType`: *ami-00c0ea23e53f48472* and *t3.small*. But **you must change** the values for `securityGroupId`, `subnetId`, `hostZone`, and `hostZoneId`, **to the values of your** security group id, subnet id, host zone (domain name) and host zone id, of which you made a note in the previous episode. + The key word `subnetId` and its value are optional. If not specified, the scripts will try to obtain a subnetID from the user’s AWS account. We have successfully tested the scripts to obtain and use a subnetID running the scripts with a personal AWS account and with an institutional AWS account. If the scripts cannot obtain a subnet-id, they will display a message describing how to obtain a subnet-id (see section [Select an AWS Subnet to Access the Instances](./01-configure-instances-internet.qmd#select-an-aws-subnet-to-access-the-instances)). If you are using an AWS institutional account, you may need to contact your IT department --- AWS allocates default subnets to each account but these may have been changed by your IT department. + + The keywords `hostZone` and `hostZoneId` and their values are optional. If specified and valid, each instance will be accessed using a domain name such as the following: `instance01.cloud-span.aws.york.ac.uk`, where `instance01` is an example of a specified instance name and `cloud-span.aws.york.ac.uk` is the base domain name (`hostZone`) in this example. If `hostZone` and `hostZoneId` and their values **are not specified**, each instance will be accessed using the public IP address or the generic domain name allocated by AWS, which will look like the following: `34.245.22.106` or `ec2-34-245-22-106.eu-west-1.compute.amazonaws.com`. -The value for `imageID`, *ami-00c0ea23e53f48472*, is the AWS resource ID of the Cloud-SPAN AMI we used for the Cloud-SPAN Genomics course. Later you may eventually need to use other AMIs to create instances, as discussed in the next episode. +- `tags.txt` --- contains a set of space-separated “key value” pairs to tag instances and +related resources upon creation. **This file is optional**. If specified, it must contain only +one “key value” pair per line. Up to 10 “key value” pairs are processed. Examples: -The value for `instanceType`, *t3.small*, specifies the type of virtualised hardware the instances you are going to create will use. The type *t3.small* has 2 processors (vCPUs) and 2 Giga Bytes of main memory. Later you may eventually need to choose smaller or greater instance types with fewer or more processors and smaller or larger main memory, as discussed in the next episode. + ```{.default filename="tags.txt - Cloud-SPAN"} + group BIOL + project cloud-span + status prod + pushed_by manual + ``` -### Configuring the file `tags.txt` -You **cannot use another name** for the file `tags.txt`. Inside the file you must specify all the following items of information: +The three files (*instancesNamesFile.txt*, `resourcesIDs.txt` and `tags.txt`) must be placed (or created) **inside** a directory called `inputs`, and the `inputs` directory must be placed **inside** within at least one other directory whose name you can choose, and to which we refer to as **Workshop Environment** (WE). We use this directory structure to manage instances for our workshops: -``` {.bash filename="Output"} -name instance -group BIOL -project cloud-span -status prod -pushed_by manual -defined_in manual +``` {.default filename="Listing"} +courses ### you can omit this directory or use other name + genomics01 ### workshop/course WE name; you can use other name + inputs ### you CANNOT use other name + instancesNames.txt ### you can use other name + resourcesIDs.txt ### you CANNOT use other name + tags.txt ### you CANNOT use other name + outputs ### created automatically by the Scripts - do not modify + genomics02 ### another WE: inputs and outputs directories inside + metagenomics01 ### another WE: inputs and outputs directories inside ``` -You **cannot change the values on the left column**. +We handle a *courses* directory that contains a WE directory for each course /workshop that we run, for example: *genomics01*, *genomics02*, *metagenomics01*, etc. Each course directory has its `inputs` directory and inside the three three files mentioned above: *instancesNames.txt*, `resourcesIDs.txt`, and `tags.txt` (note that we are using *this style* for file/directory names you can choose and `this style` for file/directory names you cannot change). + +You must create the directory structure above, or a similar one, and those three files before running the Scripts to create the instances for a course. -You can change the values on the right column. When creating and tagging each instance /resource, the Scripts change the value for the tag `name` (instance) to the actual name of each instance or with something like *resource-for-instanceName*. For this course, we recommend to use the values shown above. +The `outputs` directory inside each WE directory is created automatically by the Scripts to save the **results** of running the Scripts. -### Running the Scripts --- overview and **constraints** +### How to run the Scripts --- overview and constraints To run any of the Scripts you only need to pass the name of your file *instancesNames.txt* as a parameter to the script you want to run. The Scripts can access the files `resourcesIDs.txt` and `tags.txt` based on the location of *instancesNames.txt*. @@ -140,50 +133,51 @@ The Scripts can access the files `resourcesIDs.txt` and `tags.txt` based on the Assuming the directory structure and file names above, the script `csinstances_create.sh` should be run as shown in these two code boxes (see notes below): ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ csinstances_create.sh courses/genomics01/inputs/instancesNames.txt -... ### RESULTS +... ### RESULTS ``` ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ cd courses/ -csuser@cloud-admin-instance:~/courses +csuser@csadmin-instance:~/courses $ csinstances_create.sh genomics01/inputs/instancesNames.txt -... ### RESULTS +... ### RESULTS ``` In the code boxes: - we are showing our **terminal prompt** to highlight the current working directory from which `csinstances_create.sh` is being run. -- our shell prompt is configured to use two lines, displaying `csuser@cloud-admin-instance: working-dir` in the first line, and the dollar sign `$` on its own on the second line. +- our shell prompt is configured to use two lines, displaying `csuser@csadmin-instance: working-dir` in the first line, and the dollar sign `$` on its own on the second line. - the ***courses*** directory was created at the home (`~`) directory - in the first code box, you are running `csinstances_create.sh` while being in the home directory; hence the file name (relative path) that you must pass to `csinstances_create.sh` is `courses/genomics01/inputs/instancesNames.txt` - in the second code box, you are first moving to the *courses* directory with the command `cd`; hence the file name (relative path) that you must pass to `csinstances_create.sh` is `genomics01/inputs/instancesNames.txt` -#### **DON'T RUN** the Scripts **with a relative path that does not include the name of the "*course/workshop*"** directory --- **they will fail** +#### **DON'T RUN** the Scripts **with a relative path that does not include the name of the "*course/workshop*" WE** directory --- **they will fail** + Following the code examples above, you should not move into the *genomics01* directory and run the scripts as shown below: ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~/courses +csuser@csadmin-instance:~/courses $ cd genomics01/ -csuser@cloud-admin-instance:~/courses/genomics01 +csuser@csadmin-instance:~/courses/genomics01 $ csinstances_create.sh inputs/instancesNames.txt -... ### FAILURE RESULTS +... ### FAILURE RESULTS ``` However, the following will work: ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~/courses/genomics01 +csuser@csadmin-instance:~/courses/genomics01 $ csinstances_create.sh ../genomics01/inputs/instancesNames.txt $ csinstances_create.sh ~/courses/genomics01/inputs/instancesNames.txt ``` -The reason to force the specification of the "course/workshop" directory name in running the Scripts is to prevent (reduce the likelihood of) running the Scripts in the wrong place, which can happen when you are managing instances for different courses or test runs at the same time. +The reason to force the specification of the "course/workshop" WE directory name in running the Scripts is to prevent (reduce the likelihood of) running the Scripts in the wrong place, which can happen when you are managing instances for different courses or test runs at the same time. #### *Running the other Scripts*: -All the Scripts are run in the same way, passing as parameter the name of your file *instancesNames.txt* as shown below. While the scripts `aws_*.sh` are meant to be run (invoked) only by the scripts `csinstances_*.sh`, **running the scripts `aws_*.sh` directly by the user** may be useful for improving them or fixing a failed step in creating multiple instances, as will be discussed later in this episode. +All the Scripts are run in the same way, passing as parameter the name of your file *instancesNames.txt* as shown below: ``` {.bash filename="Code"} $ csinstances_stop.sh genomics01/inputs/instancesNames.txt @@ -194,70 +188,100 @@ $ aws_domainNames_create.sh genomics01/inputs/instancesNames.txt $ aws_loginKeyPair_delete.sh genomics01/inputs/instancesNames.txt ``` +While the scripts `aws_*.sh` are meant to be run (invoked) only by the scripts `csinstances_*.sh`, **running the scripts `aws_*.sh` directly by the user** may be useful for improving them or fixing a failed step in creating multiple instances, as will be discussed later in the section [Troubleshooting](#troubleshooting) below. + + # 2. Configure the Scripts Running Environment -This and the subsequent sections are hands-on. In this section you are going to configure the Scripts running environment for a sample course/workshop --- in subsequent sections you will use this environment to create and manage instances. Your are going to: +This and the subsequent sections are hands-on. In this section you are going to configure the Scripts running environment for a sample course. You will use this environment in subsequent sections to create and manage instances. In this section you are going to: -- create the directory structure `courses/instances-management/inputs` at the home directory +- create the workshop environment (WE) directory structure `courses/instances-management/inputs` at the home directory - create the three configuration files: *instancesNames.txt*, `resourcesIDs.txt` and `tags.txt` +- configure the AWS CLI (command line interface) --- *if you are attending a workshop **using a Cloud-SPAN AWS account*** While you can choose a different name for the file *instancesNames.txt* and for the parent directories of the `inputs` directory, for this course we recommend that you use the names suggested as otherwise you will have to adapt the intructions below. -### Create the directory structure: `courses/instances-management/inputs` +### Create the WE directory structure Open your (Git Bash, Linux, Mac, or AWS CloudShell) terminal where you installed the Scripts. Being at the home directory (`~`), type the `mkdir` command below and press Enter to create the directory structure: ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ mkdir -p courses/instances-management/inputs ``` -The option `-p` (or `--parents`) tells `mkdir` to create parent directories as needed, and not to complain if any of the directories exist. +The option `-p` (or `--parents`) tells `mkdir` to create parent directories as needed, and not to complain if any of those directories exist. + +### Create the files: *instancesNames.txt*, `resourcesIDs.txt` and `tags.txt` +You will normally use your preferred editor to create those files. However, to simplify things somewhat you are going to use (copy) the files that we have prepared beforehand. + +The commands to copy those files to your WE are inside in one of the bars below. Click on the bar that corresponds to "*how you are taking this course*" to display the commands you will type: + +::: {.callout-note collapse="true"} +## You are attending a workshop **using a Cloud-SPAN AWS** account +You must be in your home directory. Copy those files to your WE `inputs` directory as follows: + +``` {.bash filename="Code"} +csuser@csadmin-instance:~ +$ cp .local/bincsaws/instances-management-course-data/inputs/* courses/instances-management/inputs +``` +::: -### Create the configuration files: *instancesNames.txt*, `resourcesIDs.txt` and `tags.txt` -You will normally use your preferred editor to create those files. However, to simplify things somewhat you are going to use the files that we have prepared beforehand, so that you will only need to edit the file `resourcesIDs.txt` to specify your domain name and the AWS resource IDs for your host zone, security group, and subnet. +::: {.callout-note collapse="true"} +## You are self-studying the course or attending a workshop using your AWS account +You must be in your home directory. The files we have prepared are in the directory `~/_tmp_cloudspan_aws/` where you downloaded the Scripts with `git clone ..` in the previous lesson. Check you have that directory with the `ls` command. -#### *Copy the files prepared for this section* +If you don't have the directory `~/_tmp_cloudspan_aws/` in your terminal environment, download the Scripts again with the `git clone` command: -The files we have prepared are in the directory where you downloaded the Scripts with `git clone ..` (in Lesson 1, Episode 3 or 4). +``` {.bash filename="Code"} +$ git clone https://github.com/Cloud-SPAN/aws-instances.git ~/_tmp_cloudspan_aws +``` -Copy the files to your course `inputs` directory as follows (note that you must be in your home directory): +Copy the files to your WE `inputs` directory as follows: ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ cp _tmp_cloudspan_aws/instances-management-course-data/inputs/* courses/instances-management/inputs -csuser@cloud-admin-instance:~ -$ ls courses/instances-management/inputs/ -instancesNames.txt resourcesIDs.txt tags.txt ``` +::: -#### *If you don't have the directory* `~/_tmp_cloudspan_aws/` -If you don't have that directory in your terminal environment, download the Scripts again with the `git` command below and then do the copy in the code box above: +### Check the contents of each file +Let's list the files you copied: ``` {.bash filename="Code"} -$ git clone https://github.com/Cloud-SPAN/aws-instances.git ~/_tmp_cloudspan_aws +csuser@csadmin-instance:~ +$ ls courses/instances-management/inputs/ ``` +```{.default filename="Output"} +instancesNames.txt resourcesIDs.txt tags.txt +``` + +Remember that the `tags.txt` file is optional but we use one throughout the examples in the course. +Let's look now at the contents of each file so you can later relate the inputs to the outputs of the Scripts. -#### *Check the contents of each file* -Let's look at the contents of each file so you can later relate the inputs to the outputs of the Scripts. Also, **you need to edit** the last four lines of the file `resourcesIDs.txt`. +#### ***instancesNames.txt*** -The contents of the file *instancesNames.txt* you copied is below. We are going to create three instances with the names: instance01, instance02, instance03. We don't need to create more instances to learn how to use the Scripts. We have used the Scripts to create up to 37 instances in one go in about 15 minutes. +The contents of the file *instancesNames.txt* you copied is shown below. We are going to create three instances with the names: instance01, instance02, instance03. We don't need to create more instances to learn how to use the Scripts. We have used the Scripts to create up to 37 instances in one go in about 15 minutes. ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ cat courses/instances-management/inputs/instancesNames.txt +``` +``` {.default filename="Output"} instance01 instance02 instance03 ``` - -The contents of the file `tags.txt` you copied is below. The instances and related resources to be created will be labelled with the tags in this file. +#### **`tags.txt`** +The contents of the file `tags.txt` you copied is shown below. The instances and related resources to be created will be labelled with the tags (*key-value*) in this file (you can use other key-value's). ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ cat courses/instances-management/inputs/tags.txt +``` +``` {.default filename="Output"} name instance group BIOL project cloud-span @@ -266,140 +290,310 @@ pushed_by manual defined_in manual ``` +#### **`resourcesIDs.txt`** +The contents of the file `resourcesIDs.txt` you copied is shown below. It contains the AWS resources to use in creating instances and related resources, which means that each instance will be: -The contents of the file `resourcesIDs.txt` you copied is below. **You need to update the values on the right column of the last four rows** with the values of your: security group id, subnet id, host zone (domain name), and host zone id --- don't change anything else. +- a copy of the Amazon Machine Image (AMI) whose `imageID` is `ami-00c0ea23e53f48472` +- running on an `instanceType` (virtual machine hardware) `t3.small` +- using the security configuration whose securityGroupId is `sg-07fde18971b673b39` +- etc. ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ cat courses/instances-management/inputs/resourcesIDs.txt -imageId ami-00c0ea23e53f48472 -instanceType t3.small -securityGroupId sg-07fde18971b673b39 -subnetId subnet-00f3dc83b7407df8c -hostZone cloud-span.aws.york.ac.uk -hostZoneId Z012538133CJ0WPYPR3UZ ``` +``` {.default filename="Output"} +imageId ami-00c0ea23e53f48472 +instanceType t3.small +securityGroupId sg-07fde18971b673b39 +subnetId subnet-00f3dc83b7407df8c +hostZone cloud-span.aws.york.ac.uk +hostZoneId Z012538133CJ0WPYPR3UZ +``` + +Recall that the AWS resources specified in a `resourcesIDs.txt` file are all validated. The value of `imageId` is validated to correspond either to an AMI in your AWS account or to a public AMI available in the AWS region on which the scripts are running --- the `imageID` value in the listing above corresponds to a **public** Cloud-SPAN AMI of size 30 GB available in the eu-west-1 (Ireland) region. The value of `instanceType` is validated to be a valid AWS instance type: `t3.small` in the listing above corresponds to an instance with 2 processors (vCPUs) and 2 Giga Bytes of main memory. + +The values of `securityGroupId`, `subnetId`, `hostZone`, and `hostZoneId` are validated to exist in the AWS account you are using to invoke the Scripts. + +### Edit the file `resourcesIDs.txt` or configure the AWS CLI +Because the values of `securityGroupId`, `subnetId`, `hostZone`, and `hostZoneId` **must exist in the AWS account you are using** to invoke the Scripts, if you are self-studying the course or attending a workshop **using your AWS account**, you must edit the **last four lines** in that `resourcesIDs.txt` file as described within the bar "Edit `resourcesIDs.txt` --- if you are using *your* AWS account" below. + +If you are attending a workshop **using a Cloud-SPAN AWS account**, the values of `securityGroupId`, etc. are valid for the Cloud-SPAN account you are using. However you need to configure the AWS CLI installed in your AWS instance as described in the within the bar "Configure the AWS CLI --- if you are .. using a *Cloud-SPAN* AWS account" below. +::: {.callout-note collapse="true"} +## Edit `resourcesIDs.txt` if you are using *your* AWS account +#### *What you need to edit*: +You need to edit the lines that correspond to `securityGroupId`, `subnetId`, `hostZone` and `hostZoneId` as follows: -By not altering the first two rows, you are going to use the Cloud-SPAN AMI, whose id is *ami-00c0ea23e53f48472*, as template to create the instances of type *t3.small* which has 2 processors (vCPUs) and 2 Giga Bytes of main memory. +- `securityGroupId` --- change the value (***second field in the line***) with the value of your security group id. +- `subnetId` --- change the value with the value of your subnet id --- or, **delete** this line and let the Scripts obtain a subnet id automatically. +- `hostZone` --- change the value with the value of your host zone (domain name) --- or, **delete** this line and let the scripts create instances without domain names. +- `hostZoneId` --- change the value with the value of your host zone id --- or, **delete** this line and let the scripts create instances without domain names. -#### *Edit the file* `resourcesIDs.txt` +You need to change the values of both `hostZone` and `hostZoneId`, or **delete** the two containing lines. + +#### *Editing the file* `resourcesIDs.txt` You need to use a **plain-text editor** such as `nano`, `vim`, `emacs`, etc., to update the file `resourcesIDs.txt` so that no special characters get into the file which may cause the Scripts to fail. The editors `nano` and `vim` are available both in the Git Bash terminal and the AWS CloudShell terminal. If you are using the AWS CloudShell terminal, you can download the file to your machine and edit it there with your preferred **plain text editor**, and upload it back to your *AWS CloudShell enviroment*. The screenshot below shows the menu to download and upload files: -![.](/images/02-instances-management/01-aws-cloudshell-actions-menu.png){width="900px" fig-alt="Screenshot of AWS Console page in a browser showing the AWS CloudShell terminal with the options Actions, Download File and Upload File circled."} +![](/images/02-instances-management/01-aws-cloudshell-actions-menu.png){width="900px" fig-alt="Screenshot of AWS Console page in a browser showing the AWS CloudShell terminal with the options Actions, Download File and Upload File circled."} +::: + +::: {.callout-note collapse="true"} +## Configure the AWS CLI if you are .. using a *Cloud-SPAN* AWS account +The AWS CLI (command line interface) is a software tool that enables you to interact with AWS through commands that can be run either on a terminal or within shell scripts. + +The Scripts invoke /run the AWS CLI to make requests to manage (create, allocate, …, and delete) AWS services such as instances, storage, etc. For such requests to be successful, the AWS CLI must be configured to use an AWS account. + +The AWS instance provided to you by the Cloud-SPAN team has already installed the Scripts and the AWS CLI, **but** the **AWS CLI is not configured** yet for security reasons. + +To configure the AWS CLI in your instance, run the command `aws configure` as shown below: + +``` {.bash filename="Code"} +csuser@csadmin-instance:~ +$ aws configure +``` + +The following four prompts will be displayed, one at a time: + +``` {.default filename="aws configure prompts"} +AWS Access Key ID []: +AWS Secret Access Key []: +Default region name []: +Default output format []: +``` + +Please **wait for** the workshop instructor to give you the **data values** to enter to each prompt. + +Once you have entered the fourth value, the AWS CLI in your instance will be ready to use. Then, check the configuration by running the following command: + +``` {.bash filename="Code"} +aws ec2 get-vpn-connection-device-types +``` + +If your configuration is correct, you should see an output like this: + +``` {.default filename="Output"} +{ + "VpnConnectionDeviceTypes": [ + { + "VpnConnectionDeviceTypeId": "36ef5d04", + "Vendor": "Barracuda", + "Platform": "NextGen Firewall F-Series", + "Software": "6.2+" + }, +... +``` + +If instead you see an authorisation failure message like the one below, your configuration is not right. Run `aws configure` again and check that you properly copy-paste the values you were given. + +``` {.default filename="Output"} +An error occurred (AuthFailure) when calling the GetVpnConnectionDeviceTypes operation: AWS was not able to validate the provided access credentials +``` +::: + # 3. Create the Instances To create the instances specified in the previous section, you only need to run `csinstances_create.sh` as shown below while being at the home directory (use the `Tab` key to complete the names of the script, the directories and the file): ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ csinstances_create.sh courses/instances-management/inputs/instancesNames.txt ``` +Once you press `Enter` the output you will see in your terminal will be like the one in the screenshot below: + +Creating instances involves **five** or **four** steps depending on whether domain names to +access instances are to be managed or not: + +1. **Validating the contents of the Scripts configuration files**: *instancesNames.txt*, `tags.txt` (if found), and `resourcesIDs.txt`. The validation is as described above and outlined in the screenshot below. The files `tags.txt` and `resourcesIDs.txt` are accessed by the Scripts based on the path of the file *instancesName.txt* passed as a parameter. If **no problem** is found in these files, the option to continue with the configuration detected, regarding managing or not managing domain names to access instances, is displayed for the user to confirm or cancel the run. + + ![](/images/02-instances-management/fig01-scripts-run-validating-config-files.png){width="900px" fig-alt="Screenshot of Linux terminal showing the terminal prompt, the run of the command csinstances_create.sh courses/instances-management/inputs/instancesNames.txt, and part of the output results of that command."} + + If **there is a problem** a problem in any of the files, messages (not shown in the screenshot) are displayed to specify the specific problem/s in each file and the run is aborted.\ -Once you press Enter the output you will see in your terminal will be like the one in the screenshot below. The script `csinstances_create.sh` will invoke each of the scripts that creates /configures AWS resources. Each of these scripts will first display in cyan colour what it does ("Creating login keys" for example); it will then display to the terminal **part of** the results of creating /configuring the resources, and save to files in the `outputs` directory all of those results. + In the screenshot above, the configuration detected corresponds to **managing domain names** to access instances; that is, `hostZone` and `hostZoneId` and valid values were specified in the `resourcesIDs.txt` file and found to be valid.\ + If `hostZone` and `hostZoneId` **are not** specified, the option to continue looks like this:\ + ```{.default filename="Option message to continue with no domain names"} + --> NO base domain name was FOUND. + Each instance to be created will be accessed with the IP address or the generic domain name provided by AWS, which look like this: 34.245.22.106 or ec2-34-245-22-106.eu-west-1.compute.amazonaws.com. + Would you like to continue (y/n)?: + ``` -![.](/images/02-instances-management/02-results-of-running-csinstance_create.sh.png){width="900px" fig-alt="Screenshot of Linux terminal showing the terminal prompt, the run of the command csinstances_create.sh courses/instances-management/inputs/instancesNames.txt, and part of the output results of that command."} + Now **Type** `y` and **press** `Enter` to continue with steps 2, 3 and 4, which are shown in the screenshot below. -In the screenshot above, you can see that **login keys** are created first because their resource ID's must be passed as parameters to create the corresponding **instances**, which are created second. Then **IP addresses** are allocated (see bottom of the screenshot above), and then the instances **domain names** are created because creating a domain name requires passing as parameter the IP address to which the domain name will be mapped. +2. **Creating the login keys**. Login keys are created first because their resource ID’s must be passed as parameters to create the corresponding instances. -Once the domain names are created, the allocated **IP addresses** are **associated to** the corresponding **instances**: +3. **Creating the instances**, each configured to use one of the login keys created in the +previous step. When an instance is created (launched for the first time), or started after having been stopped, AWS randomly allocates an IP address to the instance. The IP address of each instance is used in step 4 below if domain names are to be managed. -![.](/images/02-instances-management/03-results-of-running-csinstance_create.sh-associating-IPaddress.png){width="900px" fig-alt="Screenshot of Linux terminal showing the middle part of the output results of the command csinstances_create.sh run previously."} +4. **Creating the instance domain names** as mapped to the respective instance IP addresses — this step **is only run** if `hostZone`, +`hostZoneId` and valid values are specified in the `resourcesIDs.txt` file. -The final step in creating instances is to **configure each instance**, see bottom of screenshot above and the screenshot below. We will look in some detail at the output of configuring instances in Section 8. Troubleshooting and in the next episode. For the time being you only need to know that configuring each instance consists of: + ![](/images/02-instances-management/fig02-scripts-run-creating-instances.png){width="900px" fig-alt="Screenshot of Linux terminal showing the terminal prompt, the run of the command csinstances_create.sh courses/instances-management/inputs/instancesNames.txt, and part of the output results of that command."} -- enabling access to the `csuser` account --- a newly created instance can only be accessed through the `ubuntu` user account -- configuring the "**host name**" to be the instance domain name (instead of the instance IP address), so that the terminal prompt, when accessing the instance with `ssh`, is something like this: `csuser@instance01:~ $` and `ubuntu@instance01.cloud-span.aws.york.ac.uk:~ $` --- instead of something like this: `csuser@52.212.13.253:~ $` and `ubuntu@52.212.13.253:~ $`. +5. **Configuring each instance**, both to enable the `csuser` account (used by workshop +participants) to be logged in and to change the instance host name to the instance name used to create the instance (the default host name is the instance IP address), see notes below. + + ![](/images/02-instances-management/fig03-scripts-run-configuring-instances.png){width="900px" fig-alt="Screenshot of Linux terminal showing the terminal prompt, the run of the command csinstances_create.sh courses/instances-management/inputs/instancesNames.txt, the instances configuration step."} + + - a newly created instance can only be accessed through the `ubuntu` user account. It is the only account that is enabled to be logged in by the AWS service that launches instances. + When an instance runs (boots) for the first time, that service adds the public key part of the login key created to access the instance to the file `/home/ubuntu/.ssh/authorized_keys`. Thereafter, the private key part of the login key can be used with `ssh` to access the `ubuntu` account. To enable the `csuser` account to be logged in in each instance, this configuring step runs a Bash script in the `ubuntu` account (in each instance) that copies that file (`../authorized_keys`) to the `csuser` account in `/home/csuser/.ssh/authorized_keys`. + - the default "***host name***" of a newly created instance is its IP address. As such, the terminal prompt when accessing an instance with `ssh` looks like this: `csuser@52.212.13.253:~ $` and `ubuntu@52.212.13.253:~ $`. This configuring step changes the default host name to the instance domain name (or the instance name if domain names are not being managed) through remotely invoking on each instance the command `sudo hostnamectl set-hostname "instance-domain-name"` (or "`instance-name`" if domain names are not being managed). Thereafter, the terminal prompt when accessing an instance with `ssh` looks like this: `csuser@instance01:~ $` and `ubuntu@instance01.cloud-span.aws.york.ac.uk:~ $`. + +Steps 2, 3, 4, and 5 are respectively performed by the scripts `aws_loginKeyPair_create.sh`, `aws_instances_launch.sh `, `aws_domainNames_create.sh`, and `aws_instances_configure.sh`. The script `csinstances_create.sh` invokes those scripts after validating the Scripts configuration files (step 1). -![.](/images/02-instances-management/04-results-of-running-csinstance_create.sh-configuring-instance.png){width="900px" fig-alt="Screenshot of Linux terminal showing the last part of the output results of the command csinstances_create.sh run previously."} +Those scripts display to the terminal only **part of** the results of creating or configuring the relevant resources, and save all of those results to files in the `outputs` directory, as described below. # 4. Check the Instances Are Accessible You should now be able to login to the instances you just created. It is convenient to login to a few of the instances that you create for a course to verify that they are reachable, and hence everything went as expected. -You can login to an instance as shown in the introduction to this lesson, running `ssh` specifying the login key, the `csuser` and the instance domain name, that is: +You can login to an instance as shown in the introduction to this lesson, running `ssh` specifying the login key with the `-i` (identity file) option, the `csuser`, and the instance domain name or instance IP address, that is: ``` {.bash filename="Code"} -$ ssh -i login-key-instance001.pem csuser@instance01.cloud-span.aws.york.ac.uk ### -i stands for identity file +$ ssh -i login-key-instance01.pem csuser@instance01.cloud-span.aws.york.ac.uk +### or +$ ssh -i login-key-instance01.pem csuser@52.212.13.253 + ``` +However, using `ssh` as shown above is adequate for the end user of each instance who will be using the same `ssh` command to access the same instance throughout a course, while being at the directory where the login key file has been stored. -However, using `ssh` as shown above is adequate for the end user of the instance who will be using the same `ssh` command to access the same instance throughout a course, while being at the directory where the login key file is. +As the **admin user** of the Scripts that manage multiple instances, at times managing two or more courses or tests simultaneously, you need to be able to login to different instances more easily. -As the manager of multiple instances, at times managing two or more courses or tests simultaneously, you need to be able to login to different instances more easily. +### Use `lginstance.sh` to login to instances -The script `lginstance.sh` will help you do that --- it was downloaded into your environment along with the Scripts. It is used specifying only the *instance login key file* and the user account to login to, either `csuser` or `ubuntu` --- both users use the same login key files. Assuming the configuration files used above, and being at the home directory, logging into *instance01* would be thus: +The script `lginstance.sh` will help you login to instances more easiyly --- it was downloaded into your environment along with the Scripts. It is used specifying only the **instance login key file** and the **user account** to login to, either `csuser` or `ubuntu` --- both users use the same login key files. Assuming the configuration files used above, and being at the home directory, logging into *instance01* would be thus: + + +::: {.column-page} + ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ -$ lginstance.sh courses/instances-management/outputs/login-keys/login-key-instance01.pem csuser +csuser@csadmin-instance:~ +$ lginstance.sh courses/instances-management/outputs/login-keys/login-key-instance01.pem csuser - lginstance.sh: logging you thus: ### this line and the next one are displayed by lginstance.sh - ssh -i courses/instances-management/outputs/login-keys/login-key-instance01.pem csuser@instance01.cloud-span.aws.york.ac.uk + lginstance.sh: logging you thus: ### this and the next line are displayed by lginstance.sh + ssh -i courses/instances-management/outputs/login-keys/login-key-instance01.pem csuser@instance01.cloud-span.aws.york.ac.uk -### Welcome message from instance01 +### Welcome message from instance01 +csuser@instance01:∼ ### instance01 prompt ``` - +::: `lginstance.sh` builds the `ssh` command you need to login to an instance and runs it. The login keys are in the directory `~/courses/instances-management/outputs/login-keys/`, so you need to specify the relative path (from your current working directory) to the corresponding login key file. That long command is rather easy to enter using the `Tab` key for the shell to complete the name of the script `lginstance.sh`, the names of the intermediate directories and the name of the login key .pem file. **Try it**. -What we found would get on the way (before developing `lginstance.sh`) was specifying the pair user@instance-domain-name (`csuser@instance01.cloud-span.aws.york.ac.uk`) because **the shell cannot help you complete it** using the `Tab` key. +What we found would get on the way (before developing `lginstance.sh`) was specifying the pair user@instance-domain-name (`csuser@instance01.cloud-span.aws.york.ac.uk`) because **the shell cannot help you complete it** using the `Tab` key (because it only complets command names and file names). + +### `lginstance.sh` works the same using or not using domain names +If you are not using domain names, as admin user you can still use `lginstance` the same way, specifying the instance **login key file** and the **user account** to login, but `lginstance` will use the instance IP address (as opposed to the instance domain name) to log you in, as shown below: + +::: {.column-body-outset} +``` {.bash filename="Code"} +csuser@csadmin-instance:~ +$ lginstance.sh courses/instances-management/outputs/login-keys/login-key-instance01.pem csuser + + lginstance.sh: logging you thus: ### this and the next line are displayed by lginstance.sh + ssh -i courses/instances-management/outputs/login-keys/login-key-instance01.pem csuser@3.253.59.74 -`lginstance.sh` extracts the domain name from the `resourcesIDs.txt` file. +### Welcome message from instance01 +csuser@instance01:∼ ### instance01 prompt +``` +::: + +If domain names are being managed, `lginstance.sh` extracts the base domain name from the file `../inputs/resourcesIDs.txt` and prepends the instance name to create the domain name of the instance to login. Otherwise, `lginstance.sh` gets the instance IP address by querying AWS using the `instanceId` stored in the instance creation results file described in the next section. + +### Not using domain names = workshop participants use IP addresses +If domain names are not being used, workshop participants will need to use the IP address of their instances to login. Hence, **admin users** need to know where the IP address of each instance is stored, which is covered in the next section, under [IP addressess for workshop participants](#ip-addressess-for-workshop-participants). + +Note that whether domain names are managed or not, the shell prompt of each instance is the instance name specified to create the instance. Thus, if an instance needs some **troubleshooting**, the workshop participant can specify the instance name, and the admin user can use `lginstance.sh` to login to the instance based on the instance name (and not the instance IP address). # 5. Understand the Results of Creating Instances -The results of creating instances that you saw displayed at the terminal, **and more**, are saved to files in the `outputs` directory. The `outputs` directory and its content are created by the Scripts as needed. Run the `ls` command below to list the contents of the `outputs` directory: +The various results of creating instances shown in the screenshots above, and **other results**, are saved to files in the `outputs` directory. The `outputs` directory and its content are created by the Scripts as needed, within the workshop environment (WE) directory being used, in our running example `~/courses/instances-management/`. + +Run the `ls` command below to list the contents of the `outputs` directory: ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ ls courses/instances-management/outputs/ ``` +The output of the `ls` command should be the list of sub-directories (in brown) in the screenshot below --- only the date and time at the end of the directory name `instances-configuration-output20240930.140046` will be different in your output: -The output of the `ls` command should be the list of sub-directories (in brown) in the screenshot below: +![](/images/02-instances-management/05-results-of-running-csinstance_create.sh-outputs-dir.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of the command ls courses/instances-management/outputs/; the results show the names of the directories created by the scripts invoked by the scripts csinstances_create.sh."} -![.](/images/02-instances-management/05-results-of-running-csinstance_create.sh-outputs-dir.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of the command ls courses/instances-management/outputs/; the results show the names of the directories created by the scripts invoked by the scripts csinstances_create.sh."} +### Results files in the `outputs` directory --- managing domain names +Each of the Scripts that creates or configures AWS resources **sends** requests to **AWS**. AWS validates and performs each request if it is valid, and **sends back** the result of the request wich states whether the request was successful or not, and if **successful**, the **resource ID** of the created resource or configuration requested along with other pieces of information. -Each of the Scripts that creates or configures AWS resources **makes/sends requests to AWS**. AWS validates and performs each request if it is valid, and **returns back** the result of the request wich states whether the request was successful or not, and if **successful**, the **resource ID** of the created resource or configuration requested along with other information. +Each of the Scripts that makes such requests creates a **sub-directory** within the `outputs` directory **to save** the results of each request, to a file whose name has, as a sub-string, the name of the relevant instance as specified in the input file *instancesNames.txt* that you use to create the instances. -Each of the Scripts that makes such requests creates a sub-directory within the `outputs` directory **to save** the results of each request, to a file whose name has, as a sub-string, the name of the relevant instance as specified in the input file *instancesNames.txt* that you use to create the instances. +The files that were created for the above run of `csinstances_create.sh` are shown in the screenshot below --- use the same `ls` commands in the screenshot to list the files in each sub-directory within your `outputs` directory. If you used the instances names we suggested to create the instances, the files created by your run of `csinstances_create.sh` should have the same names as in the screenshot: -The files that were created for the above run of `csinstances_create.sh` are shown in the screenshot below --- use the same `ls` commands in the screenshot to list the files in each sub-directory within your `outputs` directory. If you used the instances names we suggested to create the instances, the files created from your run of `csinstances_create.sh` should have the same names as in the screenshot: +![](/images/02-instances-management/06-results-of-running-csinstance_create.sh-outputs-dir-files.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of the command ls for each of the subdirectories shown in the previous screenshot; the results of each ls command shows the names of the files created by the scripts invoked by the scripts csinstances_create.sh."} -![.](/images/02-instances-management/06-results-of-running-csinstance_create.sh-outputs-dir-files.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of the command ls for each of the subdirectories shown in the previous screenshot; the results of each ls command shows the names of the files created by the scripts invoked by the scripts csinstances_create.sh."} +### Each instance name is the key to access each instance results files in the `outputs` directory -### *Each instance name is the key to access each instance results files in the* `outputs` *directory* +Once an AWS resource is created, **any further request on the resource** to, for example, configure it, stop it, .., or delete it, **must include the resource ID** received when the resource was created. -Once an AWS resource is created, **any further request on the resource** to, for example, configure it, stop it, .., and delete it, **must include the resource ID** received when the resource was created. +The naming convention of the results files mentioned above, that is, to include the name of the corresponding instance in each results file name, enables the Scripts to identify the files related to each instance to **extract the resource IDs** in order to further manage an instance and its related resources (login key, domain name), through making the corresponding requests to AWS. -The naming convention of the results files mentioned above, that is, to include the name of the corresponding instance in each file name, enables the Scripts to identify the files related to each instance and **extract the resource IDs** in order to manage the instance and its login key, domain name, etc., through making further requests to AWS. +The Scripts get the instances names to build the corresponding results files names from the + *instancesNames.txt* received as a parameter (recall that all the Scripts are run in the same way). -The two screenshots below show the resource IDs of instance01 and its IP address that are extracted to further manage the instance and its IP address. We will look in more detail at the contents of the results files in the next episode. +The screenshot below shows the resource ID of instance01 that is extracted by the Scritps to further manage the instance. Login keys IDs look like this: `"KeyPairId": "key-04b589569b2952ed2"`. -![.](/images/02-instances-management/07-results-of-running-csinstance_create.sh-instance01-id.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of the command more courses/instances-management/outputs/instances-creation-output/instance01.txt which displays the contents of that file, with the InstanceID field and its value circled."} +![](/images/02-instances-management/07-results-of-running-csinstance_create.sh-instance01-id.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of the command more courses/instances-management/outputs/instances-creation-output/instance01.txt which displays the contents of that file, with the InstanceID field and its value circled."} -![.](/images/02-instances-management/08-results-of-running-csinstance_create.sh-eip-allocation-id.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of the command more courses/instances-management/outputs/ip-addresses-allocation-output/elastic-IPaddress-for-instance01.txt which displays the contents of that file, with the AllocationID field and its value circled."} +### Results files in the `outputs` directory --- NOT managing domain names +### IP addressess for workshop participants +When domain names are **not** managed: + +- the sub-directory `domain-names-creation-output` is **not** created. +- the sub-directory `instances-creation-output` is used to store a file for each created instance that contains the IP address allocated by AWS to the instance. The name of each file is "*instance-name*`-ip-address.txt`", where "*instance-name*" is the name of the instance as specified in the *instancesNames.txt* file used to run the Scripts. + +If the three instances created in our running example had been created with **no** domain names, the files in the subdirectory `instances-creation-output` would be the ones shown in the screenshot below: + +![](/images/02-instances-management/08-no-domain-names-IP-addresses-results-files.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of the command ls for the subdirectory-instances-creation-output-no-domain-names."} + +In the screenshot, the files “instance??.txt” are created by the script `aws_instances_launch.sh`; the files “instance??`-ip-address.txt`” are created by the script `aws_instances_configure.sh`. This is because an instance public IP address is not available in the results from invoking +`aws_instances_launch.sh` (that is, in the files “*instance??*.txt”), but is available until each +instance is actually running, a condition that the script `aws_instances_configure.sh` waits +for and detects in order to recover each instance IP address to start configuring the +instance, and finally save the IP address into the file "*instanceName*`-ip-address.txt`". + +Note that, if instances are **stopped** and eventually **started**, through running `csinstances_stop.sh` and `csinstances_start.sh`, the IP address of each instance will change because AWS randomly allocates IP addresses when instances are launched for the first time or when +they are started after having been stopped. Therefore, `csinstances_start.sh` overwrites the +contents of the files “*instanceName\**`-ip-address.txt`” with the newly allocated IP +addresses. # 6. Typical Instances Life Cycle: create, stop, start and delete instances When you create instances through running `csinstances_create.sh`, the instances and their related resources (login key files, etc.) are created and configured first; then the instances are run (launched/started) and are left running until you either stop them or delete them. -Our typical scenario in managing instances is as follows. When a workshop is scheduled for delivery, say on day D, we usually send the workshop participants the instructions to login to their instances, along with their login key file, one or a few days before D. Thus we usually do the following: +Our typical scenario in managing instances is as follows. When a workshop is scheduled for delivery, say on day D, we usually send the workshop participants the instructions to login to their instances, along with their login key file, one or a few days before D. We usually follow these steps: + - **create** the instances before D for the login key files to be created. - **stop** the instances shortly after (within minutes) to reduce costs. - re-**start** the instances again some time before the workshop starts. - **delete** all the instances once the workshop is over. -We perform those four tasks running the Scripts `csinstances_*.sh` in our terminal, **but we usually are logged in** to the **AWS Console** too, at the **EC2 --- Instances** page, to check that the **instances state** changes according to the state intended by each script, as outlined below. +We perform those four tasks running the Scripts `csinstances_*.sh` in our terminal: + +- `csinstances_create.sh` +- `csinstances_stop.sh` +- `csinstances_start.sh` +- `csinstances_delete.sh` + +But **we are usually logged in** to the **AWS Console** too, at the **EC2 --- Instances** page, to check that the **instances state** changes according to the state intended by each script, as described below. Follow along: + - login to the AWS Console (with your IAM user account), type EC2 (for Elastic Compute Cloud) in the AWS search box at the top and press Enter, and then click on **Instances** on the left menu pane, see the page below. - run the scripts below in your terminal @@ -407,104 +601,154 @@ Follow along: **Only run** `csinstances_create.sh` **if you did not run it in the previous section**!!! ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ csinstances_create.sh courses/instances-management/inputs/instancesNames.txt ``` - Once you have run `csinstances_create.sh`, the instances state in the AWS Console will be **Running** as shown below (you may need to refresh the page): -![Screenshot of AWS Console EC2 Instances page in a browser showing the state, Running, of three instances previously created, circled.](/images/02-instances-management/09-ec2-instances-screen-showing-created-instances-running.png){width="900px" fig-alt=""} +![](/images/02-instances-management/09-ec2-instances-screen-showing-created-instances-running.png){width="900px" fig-alt="Screenshot of AWS Console EC2 Instances page in a browser showing the state, Running, of three instances previously created, circled."} + +Once the instances are created, you will be able to send the instances login details to workshop participants, that is: the login key files and either the instance domain names or the instance IP addresses (if domain names are not being managed). + +The login key files are in the directory `../outputs/login-keys/` (within your courses/workshop-name directory --- in our running example the *instances-management* directory). We usually upload the `login-keys` directory to a shared GDrive directory and inform someone we have done so. Someone then emails each login key file and the corresponding instance domain name to each workshop participant. Instance domain names are all saved to a GDrive file which is generated as described below. + +#### **If domain names are being managed**: +Cloud-SPAN courses are run using instance domain names. The files `../outputs/domain-names-creation-output/domain-name-create-`*instancename?..?*`.txt` contain each the domain name of an instance in the last line. However, we create a file containing all the domain names through (1) making a copy of the *instancesNames.txt* file used to create the instances, and (2) editing the file to **append** each line (instance name) with the base domain name in the `resourcesIDs.txt` file (`cloud-span.aws.york.ac.uk` in our running example). We then upload this file to a shared GDrive directory and inform someone we have done so, etc. + +#### **If domain names are NOT being managed**: +The files `../outupts/instances-creation-output/`*instancename?..?*`-ip-address.txt` contain each the IP address of an instance only. Cloud-SPAN does not follow not using domain names, but we would copy all these files to another directory and upload this directory, along with login keys files directory, to a GDrive directory and inform someone we have done so. Someone could then email each workshop participant the corresponding login key file and instanceName`-ip-address.txt` file as attachments. It should be easy to identify each pair of these files as their names contain the corresponding instance name. -Once all the instances are created, you will be able to send the login key files to whoever is responsible for emailing the workshop participants. The login key files are in the directory `../outputs/login-keys/` (within your course/workshop-name directory --- *instances-management* directory in the running example). We usually upload the `login-keys` directory to a shared GDrive directory and inform someone we have done so. +**Please note**: if you are **not** managing domain names, you may **NOT** want to **stop** and **start** the instances as suggested above and shown below, but rather to delay creating instances as much as possible until just before the start of a workshop. If you stop and start the instances, their IP address will change and you will need to email the new IP addresses to workshop participants again. Creating 30 instances takes only 10 to 15 minutes. The issue is preparing the login details of each instance (login key and IP address files) to email them to each workshop participant. ## Stopping the instances shortly after (within minutes) they are created ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ csinstances_stop.sh courses/instances-management/inputs/instancesNames.txt ``` -``` {.bash filename="Output"} +``` {.default filename="Output"} csinstances_stop.sh is stopping instances specified in input file courses/instances-management/inputs/instancesNames.txt Stopping instances: Creating directory to hold the results of stopping instances: -courses/instances-management/outputs/instances-stop-output20230101.182907 +courses/instances-management/outputs/instances-stop-output20241018.131032 Success stopping instance: instance01 +Success deleting domain: instance01.cloud-span.aws.york.ac.uk ip: 52.49.100.76 Success stopping instance: instance02 +Success deleting domain: instance02.cloud-span.aws.york.ac.uk ip: 54.217.178.245 Success stopping instance: instance03 +Success deleting domain: instance03.cloud-span.aws.york.ac.uk ip: 34.252.179.148 ``` - Once you run `csinstances_stop.sh`, the instances state in the AWS Console will change from **Running** to **Stopping** as shown below, and after a short while the state will change to **Stopped**, and will remain so until you run another script that changes the state of the instances. -![.](/images/02-instances-management/10-ec2-instances-screen-showing-created-instances-stopping.png){width="900px" fig-alt="Screenshot of AWS Console EC2 Instances page in a browser showing the state, Stopping, of three instances previously created and running, circled."} +![](/images/02-instances-management/10-ec2-instances-screen-showing-created-instances-stopping.png){width="900px" fig-alt="Screenshot of AWS Console EC2 Instances page in a browser showing the state, Stopping, of three instances previously created and running, circled."} + +Note in the output of `csinstances_stop.sh` that each instance domain name is deleted after stopping each instance. This is done because as mentioned above the IP address of each instance will change on re-starting each instance. Hence there is no reason to keep instance domain names that are mapped to IP addresses that are invalid as a result of having stopped the corresponding instances. ## Re-starting the instances just before the workshop ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ csinstances_start.sh courses/instances-management/inputs/instancesNames.txt ``` -``` {.bash filename="Output"} +``` {.default filename="Output"} csinstances_start.sh is starting instances specified in input file courses/instances-management/inputs/instancesNames.txt Starting instances: Creating directory to hold the results of starting instances: -courses/instances-management/outputs/instances-start-output20230101.185724 +courses/instances-management/outputs/instances-start-output20241018.131906 Success starting instance: instance01 Success starting instance: instance02 Success starting instance: instance03 -``` +Checking each instance is running to get its IP address: +.- instance instance01 state 16 (running) + +Please note: the IP address (52.210.84.174) of instance instance01 was saved to the file: +courses/instances-management/outputs/instances-creation-output/instance01-ip-address.txt +..- instance instance02 state 16 (running) + +Please note: the IP address (34.241.237.54) of instance instance02 was saved to the file: +courses/instances-management/outputs/instances-creation-output/instance02-ip-address.txt +..- instance instance03 state 16 (running) + +Please note: the IP address (63.35.198.137) of instance instance03 was saved to the file: +courses/instances-management/outputs/instances-creation-output/instance03-ip-address.txt + +Mapping each instance domain name to its IP address: +Success mapping domain: instance01.cloud-span.aws.york.ac.uk, ip: 52.210.84.174 +ssh-keygen -f /home/csuser/.ssh/known_hosts -R instance01.cloud-span.aws.york.ac.uk +# Host instance01.cloud-span.aws.york.ac.uk found: line 26 +/home/csuser/.ssh/known_hosts updated. +Original contents retained as /home/csuser/.ssh/known_hosts.old +Success mapping domain: instance02.cloud-span.aws.york.ac.uk, ip: 34.241.237.54 +ssh-keygen -f /home/csuser/.ssh/known_hosts -R instance02.cloud-span.aws.york.ac.uk +# Host instance02.cloud-span.aws.york.ac.uk found: line 27 +/home/csuser/.ssh/known_hosts updated. +Original contents retained as /home/csuser/.ssh/known_hosts.old +Success mapping domain: instance03.cloud-span.aws.york.ac.uk, ip: 63.35.198.137 +ssh-keygen -f /home/csuser/.ssh/known_hosts -R instance03.cloud-span.aws.york.ac.uk +# Host instance03.cloud-span.aws.york.ac.uk found: line 28 +/home/csuser/.ssh/known_hosts updated. +Original contents retained as /home/csuser/.ssh/known_hosts.old +Checking each domain is reachable: +.......... : instance01.cloud-span.aws.york.ac.uk status INSYNC (reachable) +. : instance02.cloud-span.aws.york.ac.uk status INSYNC (reachable) +. : instance03.cloud-span.aws.york.ac.uk status INSYNC (reachable) + +Please note: you may need to wait up to one hour to be able to access re-started +instances with "ssh". This is because Domain Name servers ask every hour for changes +to the configuration of instance domain names and IP addresses have just changed. +``` Once you run `csinstances_start.sh`, the instances state in the AWS Console will change from **Stopped** to **Pending** (momentarily), and then to **Running** and will remain so until you run another script that changes the instances state. -## Deleting the instances and all their resources once the workshop is over +Note in the output of `csinstances_start.sh` that all instances are started first. Instance IP addresses are then saved each to the file `../outputs/instances-creation-output/`*instanceName*`-ip-address.txt`. If domain names are **not** being managed, `csinstances_start.sh` will finish at this point. Otherwise, if domain names **are being** managed, `csinstances_start.sh` will create each instance domain name as mapped to the new IP address of each instance, and will wait (check) until each domain name status is INSYNC (reachable). + +Please note the message in the **last paragraph** of the output of `csinstances_start.sh` which is self-explanatory. (If domain names are **not** being managed, instances can be accessed with `ssh` immediately after their IP addresses have been saved.) + +## Deleting instances and related resources once a workshop is over ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ csinstances_delete.sh courses/instances-management/inputs/instancesNames.txt ``` -``` {.bash filename="Output"} +``` {.default filename="Output"} csinstances_delete.sh is terminating instances specified in input file courses/instances-management/inputs/instancesNames.txt Terminating instances: -Creating directory to hold the results of deleting instances: +Creating directory to hold the results of deleting instances: +courses/instances-management/outputs/instances-delete-output20241018.154603 Success terminating instance: instance01 -.. +Success terminating instance: instance02 +Success terminating instance: instance03 Deleting login key pairs: +Creating directory to hold the results of deleting the login keys: +courses/instances-management/outputs/login-keys-delete-output20241018.154607 Success deleting login-key: login-key-instance01; instance: instance01 -.. +Success deleting login-key: login-key-instance02; instance: instance02 +Success deleting login-key: login-key-instance03; instance: instance03 Deleting domain names: -Success deleting domain: instance01.cloud-span.aws.york.ac.uk; ip: 99.80.179.133 -.. -Disassociating following elastic IP addresses: -Success disassociating elasticIP, instance: instance01; eipAssociationId: eipassoc-0ff7c16e5ff5f7178 -.. -Deallocating elastic IP addresses: -Success deallocating eip: 99.80.179.133; instance: instance01; eipAllocationId: eipalloc-0ccacb1c92d26ec7f -.. +Creating directory to hold the results of deleting domain names: +courses/instances-management/outputs/domain-names-delete-output20241018.154610 +Success deleting domain: instance01.cloud-span.aws.york.ac.uk ip: 52.210.84.174 +Success deleting domain: instance02.cloud-span.aws.york.ac.uk ip: 34.241.237.54 +Success deleting domain: instance03.cloud-span.aws.york.ac.uk ip: 63.35.198.137 ``` - You can delete instances in any state --- you don't need to stop instances to delete them. -Once you run `csinstances_delete.sh`, the instances state in the AWS Console will change from **Stopped** or **Running** to **Shutting-down**, and after some minutes to **Terminated**, and after some more minutes the terminated instances will no longer be shown in the AWS Console. +Once you run `csinstances_delete.sh`, the instances state in the AWS Console will change from **Stopped**, or **Running**, to **Shutting-down**, and after some minutes to **Terminated**, and after some more minutes the terminated instances will no longer be shown in the AWS Console. + +The output of `csinstances_delete.sh` shows these main steps: -The output box above shows only the main steps of `csinstances_delete.sh` in deleting instances and related resources, namely: - Terminating instances - Deleting login key pairs - Deleting domain names -- Disassociating elastic IP addresses and -- Deallocating elastic IP addresses -You should always see a **Success ...** message for each resource that is terminated, deleted or deallocated, but not always for **disassociating elasticIP**s. Instead, you may see the error in the callout below. There is nothing to worry about this message, ignore it. But you should not ignore an error in any other step --- which should not happen if your configuration is valid; the Scripts have not failed since we got them right. - -::: callout-note -## "Error disassociating elasticIP, instance: instance01; ...". -This error may happen because instances are deleted first, and by the time the Script that disassociates IP addresses from instances is run, some of the associations of IP addresses to instances may no longer be valid/existant. This is more likely to happen if an instance is stopped when `csinstances_delete.sh` is run. If an instance is running, shutting it down and then terminating it will take longer, and hence its association of IP address to instance is more likely to be valid when it is to be disassociated, which will result in successfully disassociating the IP address. -::: +You should always see a **Success ...** message for each resource that is terminated or deleted. If you don't, you are probably running the Scripts in the wrong directory or with the wrong *instancesNames.txt* file. The Scripts have not failed since we got them right. ### The results of stopping, starting and deleting instances are also saved to files @@ -513,82 +757,66 @@ The results of running `csinstances_stop.sh`, `csinstances_start.sh` and `csinst Run the `ls` command below to look at the sub-directories that were created after running those scripts. ``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ +csuser@csadmin-instance:~ $ ls courses/instances-management/outputs/ ``` - The output of `ls` should be similar to this one: -![.](/images/02-instances-management/11-terminal-listing-of-outpus-dir-after-stopping-starting-deleting-instances.png){width="900px" fig-alt="Screenshot of Linux terminal showing the output of the command ls courses/instances-management/outputs/ after stopping, re-starting and deleting the three instances; the output shows the new directories created by the scripts that stopped, started and deleted the instances."} +![](/images/02-instances-management/11-terminal-listing-of-outpus-dir-after-stopping-starting-deleting-instances.png){width="900px" fig-alt="Screenshot of Linux terminal showing the output of the command ls courses/instances-management/outputs/ after stopping, re-starting and deleting the three instances; the output shows the new directories created by the scripts that stopped, started and deleted the instances."} The newly created sub-directories are those in boxes; their names are suffixed with the date and time when the relevant script was run. Inside each sub-directory there is a file for each request made by the relevant script, one request for each instance or related resource. We won't look into these files. Their main use is to register the results of AWS requests in case of errors happening. They were essential while developing the Scripts to get them right. -## Create instances in two steps to avoid unnecessary costs -You can avoid some of the cost of deploying instances by "creating instances" **in two steps**. You will first run the script `csinstances_create1stPart.sh`, and a few days later (just before the relevant workshop starts) the script `csinstances_create2ndPart.sh`. As shown below, you run those two scripts in the same way as you run all the other scripts, passing in the "instancesNamesFile" that contains the names of the instances to create: - -``` {.bash filename="Code"} -csuser@cloud-admin-instance:~ -$ csinstances_create1stPart.sh courses/instances-management/inputs/instancesNames.txt - -##### and a few days later: -csuser@cloud-admin-instance:~ -$ csinstances_create2ndPart.sh courses/instances-management/inputs/instancesNames.txt -``` - - -Doing so will save a significant cost if you are running a workshop that requires relatively large instances in terms of compute and storage capacity, up to about US $300 per day (as at 20230414), for 50 instances of type t3.2xlarge, each instance with 240GB storage. The savings come out as follows. Recall that the script `csinstances_create.sh` first runs the script that creates the **login keys**, then the script that creates the **instances**, then .. the **IP addresses** and finally .. the **domain names**. Of all these resources, it is the instances that are more costly. The script `csinstances_create1stPart.sh` only creates the login keys, the IP addresses and the domain names. You will then be able to send the workshop participants their login key and instance domain name, say, on a Friday. On the following Monday you will then run `csinstances_create2ndPart.sh` to create the instances, associate the instances to the IP addresses, and to configure the instances. - # 7. Unforseen Instance Management -The Scripts use-case scenario outlined above is typical of short workshops that last 1-4 days in a row. Yet sometimes it doesn't go that smoothly. Sometimes you won't be able to run the scripts `csinstances_*.sh` using the same file "instancesNamesFile". - -Sometimes you will receive requests like those in the list below after you have both created an "instancesNamesFile" and run the script `csinstances_create.sh` with that file, that is: some instances have been created, and then, you receive requests like: +The Scripts use-case scenario outlined above is typical of short workshops that last 1-4 days in a row. Yet sometimes it doesn't go that smoothly. Sometimes you will receive requests like those in the list below after you have both created an "instancesNamesFile" and run the script `csinstances_create.sh` with that file, that is: some instances have been created, and then, you receive requests like: - can you create N more instances? --- We received N last-minute registrations to the workshop. - can you delete instance A? --- The participant is ill and cancelled. - can you leave the instances B and C running after the workshop until *date*? --- The participants have not yet managed to cover the course but are very keen to do it. - can you stop instance D (don't delete it) until further notice? --- The participant has started the workshop but something urgent happened and will get back to it asap. -If you are to fulfill such requests, then you **need to manage multiple** "instancesNamesFile"s. This is because the Scripts create (configure, stop, start or delete) **all the instances** whose names you specify in an "instancesNamesFile" you pass in to the Scripts. +If you are to fulfill such requests, you **need to manage multiple** "*instancesNamesFile*"s. You cannot use a single "*instancesNamesFile*" because the Scripts create (configure, stop, start or delete) **all the instances** whose names you specify in the "*instancesNamesFile*" you pass to the Scripts. -For **example**, suppose the following scenario: (**1**) you have created the three instances as instructed in Section [3 Create the Instances](#3-create-the-instances) --- you used the file *instancesNames.txt*  to create the instances named *instance01*, *instance02*, and *instance03*; (**2**) you are then asked to create another instance (for a latecomer to the workshop you are running); and (**3**) for the new instance you decide to use the name *instance04*. +For **example**, suppose the following scenario: (**1**) you have created the three instances as instructed in Section 3 [Create the Instances](#create-the-instances) --- you used the file *instancesNames.txt*  to create three instances with the names *instance01*, *instance02*, and *instance03*; (**2**) you are then asked to create another instance (for a latecomer to the workshop you are running); and (**3**) for the new instance you decide to use the name *instance04*. Now you need to create another "instancesNamesFile", say, *instancesNames-04.txt*, that contains only the instance name *instance04*, and then run `csinstances_create.sh instancesNames-04.txt`. ## Avoid modifying an "instancesNamesFile" that you have already used to create intances -In the example above, you **should not add** the instance name *instance04*  to the file *instancesNames.txt*  (that you previously used to create *instances01*, *..02* and  *..03*), and then **run again** `csinstances_create.sh instancesNames.txt` --- **it will fail** as shown in the screenshot below: +In the example above, you **should not add** the instance name *instance04*  to the file *instancesNames.txt*  (that you previously used to create *instances01*, *..02* and  *..03*), and then **run again**: -![.](/images/02-instances-management/13-results-of-running-csinstance_create.sh-AGAIN-aborted.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of csinstance_create.sh and the abort message for having found that some instances or related resources have already been created."} +`csinstances_create.sh instancesNames.txt` --- **it will fail** as shown in the screenshot below: + +![](/images/02-instances-management/13-results-of-running-csinstance_create.sh-AGAIN-aborted.png){width="900px" fig-alt="Screenshot of Linux terminal showing the run of csinstance_create.sh and the abort message for having found that some instances or related resources have already been created."} ## The Scripts *check*  instance operations are *valid*  given the state of the running environment -Recall (from Section [3 Create the Instances](#3-create-the-instances)) that login keys are created first, then instances, etc. Login keys are created by the script `aws_loginKeyPair_create.sh` which, as you can see in the screenshot above, first checks whether the login key files for the instances to be created exist or not, **aborting** its execution if any such file exists. In the run shown in the screenshot, `aws_loginKeyPair_create.sh` found the login key files for *instances01*, *..02*,  and *..03*, and hence displayed the message shown and aborted its execution which, in turn, caused `csinstances_create.sh` to abort its execution as well. +Recall (from Section 3 [Create the Instances](#create-the-instances)) that login keys are created first, then instances, etc. Login keys are created by the script `aws_loginKeyPair_create.sh` which, as you can see in the screenshot above, first checks whether the login key files for the instances to be created exist or not, **aborting** its execution if any such file exists. In the run shown in the screenshot, `aws_loginKeyPair_create.sh` found the login key files for *instances01*, *..02*,  and *..03*, and hence displayed the message shown and aborted its execution which, in turn, caused `csinstances_create.sh` to abort its execution as well. As stated in that message, the files that were found contain the **AWS resource IDs** of login keys **already created**. Those resource IDs are needed to delete the login keys from your AWS account. The Scripts **do not overwrite** files that contain such resources IDs. If those files are **overwritten**, or lost, **before you delete** the corresponding instances and related resources (running `csinstances_delete.sh`), **you will need to use the AWS Console to delete manually** the corresponding instances, login keys, IP addresses and domain names. -If you want to create instances **with names** you **have already used**, you **must first delete** the instances already created and their related resources (running `csinstances_delete.sh`), then delete or rename the `../outputs/` parent directory, and finally create the instances. We have done so only for some testing, see the Troubleshooting section. +If you want to create instances **with names** you **have already used**, you **must first delete** the instances already created and their related resources (running `csinstances_delete.sh`), then delete or rename the `../outputs/` directory, and finally create the instances. We have done so only for some testing, see the [Troubleshooting](#troubleshooting) section. -All the Scripts that **create** AWS resources carry out a similar **check** to that carried out by `aws_loginKeyPair_create.sh`. For each instance name in the "instancesNamesFile" passed in to **`csinstances_create.sh`**, each script checks whether the instance, or the resource for that instance, has been created or not, which amounts to checking whether the file with the corresponding AWS resource ID exists or not. If any such file **does** exist, the script aborts its execution **without** issuing any AWS request to create resources. +All the Scripts that **create** AWS resources carry out a similar **check** to that carried out by `aws_loginKeyPair_create.sh`. For each instance name in the "instancesNamesFile" passed to **`csinstances_create.sh`**, each script checks whether the instance, or the resource for that instance, has been created or not, which amounts to checking whether the file with the corresponding AWS resource ID exists or not. If any such file **does** exist, the script aborts its execution **without** issuing any AWS request to create resources. All the Scripts that **delete** AWS resources carry out a similar check but **act on the opposite**. ... If any such file **does not** exist, the script aborts its execution **without** issuing any AWS request to delete resources. The script `aws_instances_configure.sh` does this check. -Given a set of instances names as specified in an "instancesNamesFile", the two checks above guarantee that all or none of the instances, and/or related resources, are created or deleted. +Given a set of instances names as specified in an "*instancesNamesFile*", the two checks above guarantee that all or none of the instances, and/or related resources, are created or deleted. -Finally, all the Scripts `aws_*.sh` check that the "instancesNamesFile" passed in to them has only one value (one instance name) in each line. Thus, if you inadvertently passed in to the Scripts the file name `resourcesIDs.txt` or `tags.txt`, which both have two values in each line, the Scripts will abort their execution. +Finally, all the Scripts `aws_*.sh` check that the "*instancesNamesFile*" passed to them has only one value (one instance name) in each line. Thus, if you inadvertently passed to the Scripts the file name `resourcesIDs.txt` or `tags.txt`, which both have two values in each line, the Scripts will abort their execution. ## Using multiple "*instanceNamesFile*"s to manage unforseen instances management requests -There is no problem with having multiple "instancesNamesFile"s in an `../inputs/` directory. +There is no problem with having multiple "*instancesNamesFile*"s in an `../inputs/` directory. -The main issue is how to name those multiple files so that you **keep track** of what happens, regarding instance management, throughout a workshop. +The main issue is how to name those multiple files so that you **keep track** of what happens, regarding instance management throughout a workshop. -The listing below illustrates the naming convention we follow to name multiple "instancesNamesFile"s to handle similar requests to the ones described in the introduction to this section. +The listing below illustrates the naming convention we follow to name multiple "*instancesNamesFile*"s to handle similar requests to the ones described in the introduction to this section. The listing shows the files in the `inputs` directory of a workshop we ran through late October and November 2022: -``` {.bash filename="Output"} +``` {.default filename="Output"} instancesNames20221028-create-301-25.txt instancesNames20221031-create-326.txt instancesNames20221031-create-327.txt @@ -605,7 +833,6 @@ resourcesIDs.txt tags.txt ``` - The workshop used instances of type *t3.2xlarge*, each instance with 8 processors and 32 GigaBytes of main memory and 240 GB of secondary storage --- not cheap. The instances were to run for 4 weeks unless the participants would not make progress which we checked weekly looking at the command history and whether relevant results files existed. We would delete instances that showed no progress was made. From the file listing above we can recall the following: @@ -615,32 +842,37 @@ From the file listing above we can recall the following: - On 1 Novemeber we received another request to create one more instance and handled it as just described to create instance329. - On 9 November we deleted instance307 (for not showing progress): 1. we made a copy of *instancesNames20221028-create-301-25.txt* onto *instancesNames20221109-delete-307.txt*. - 2. opened the latter with our text editor and deleted all lines except the line with the instance name instance307. + 2. opened the copy with our text editor and deleted all lines except the line with the instance name instance307. 3. we then run `csinstances_delete.sh courses/../inputs/instancesNames20221109-delete-307.txt`. -The other "instancesNamesFile"s in the list were handled similarly to the way the event on 9 November was handled, namely: +The other "*instancesNamesFile*"s in the list were handled similarly to the way the event on 9 November was handled, namely: + - creating a new file containing only the names of the target instances. - naming the file *instancesNames***Date**-**operation**-**instancesNumbersList**.txt --- where **operation** is create, stop, start, or delete. - running the relevant `csinstances_*.sh` script to create, stop, start or delete the target instances. -The tedious part of this file naming convention is specifying the **intancesNumbersList**. However, it is not required often and it gives you an accurate overview of the instances operations you have performed. +The tedious part of this file naming convention is specifying the **intancesNumbersList**. However, it is not required often and it gives you an accurate overview of the instance operations you have performed for a workshop. # 8. Troubleshooting -This last section presents the problems we have come across using the Scripts and how we solved them. +This last section presents the problems we have come across in creating instances with the Scripts and how we solved them. The problems are presented below under the error messages displayed when the problems occur, namely: + +- **Connection timed out** or **Connection refused** messages persist +- **WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED** -### "*Connection timed out*" or "*Connection refused*" messages persist +### "**Connection timed out**" or "**Connection refused**" messages persist Those messages are related to accessing an instance with `ssh`. We have come across those messages in two scenarios: -- when an instance is to be configured --- recall that *instance configuration* is the last step in creating instances; and -- when an instance is to be accessed by the end user (after the instance has been configured) +- when an instance is to be configured --- recall that *instance configuration* is the last step in creating instances. We will refer to this scenario as **During configuration scenario**. +- and when an instance is to be accessed by the end user (after the instance has been configured). We will refer to this scenario as **After configuration scenario**. -**The first scenario** is shown below. It has to do with the SSH server in the instance not yet being ready to accept any login request, and hence the client (the script `aws_instances_configure.sh` which is invoked last by `csinstances_create.sh`) receives an unsuccessful response and displays either the message "Connection timed out" or "Connections refused", or both. This situation is "normal" because we know the instance has just been launched. It's only a matter of waiting for the SSH server to be ready. +##### **During configuration scenario** +This scenario is shown below. It has to do with the SSH server in the instance not yet being ready to accept any login request, and hence the client (the script `aws_instances_configure.sh` which is invoked last by `csinstances_create.sh`) receives an unsuccessful response and displays either the message **Connection timed out** or **Connections refused**, or both. This situation is **normal** because we know the instance has just been launched. It's only a matter of waiting for the SSH server to be ready. -![.](/images/02-instances-management/12-results-of-running-csinstance_create.sh-configuring-instance2.png){width="900px" fig-alt="Screenshot of Linux terminal showing the last part of the output of running the command csinstances_create.sh; that part corresponds to the configuration step of each instance and the screenshot shows the Connection timed out and Connection refused messages circled that may arise during the configuration step."} +![](/images/02-instances-management/12-results-of-running-csinstance_create.sh-configuring-instance2.png){width="900px" fig-alt="Screenshot of Linux terminal showing the last part of the output of running the command csinstances_create.sh; that part corresponds to the configuration step of each instance and the screenshot shows the Connection timed out and Connection refused messages circled that may arise during the configuration step."} -Note in the screenshot the message in brown "**Please wait for SSH server (you may see a few 'Connection timed out/Connection refused' messages)**", and the `ssh`-related message in white just below "**ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no -i courses/...**". The script `aws_instances_configure.sh` prints those messages and then gets into a loop to repeatedly issue that `ssh` command until it is successful. In the screenshot, three issues of that `ssh` command were unsuccessful and the message "**ssh: connect to host instance01... port 22: Connection refused**" was displayed three times. Once that `ssh` command is successful, `aws_instances_configure.sh` proceedes to issue the `ssh` commands that configures the instance. That is, the sole purpose of the `ssh` command that is issued within the loop is to ensure the SSH server in the instance is ready. +Note in the screenshot the message in brown "**Please wait for SSH server (you may see a few 'Connection timed out/Connection refused' messages)**", and the message in white just below "**ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no -i courses/...**". The script `aws_instances_configure.sh` prints those two messages and then gets into a loop to repeatedly issue that `ssh` command until it is successful. In the screenshot, three issues of that `ssh` command were unsuccessful and the message "**ssh: connect to host instance01... port 22: Connection refused**" was displayed three times. Once that `ssh` command is successful, `aws_instances_configure.sh` proceedes to issue the `ssh` commands that configures the instance. That is, the sole purpose of the `ssh` command that is issued within the loop is to ensure the SSH server in the instance is ready. **How many "Connection timed out/Connection refused" (unsuccessful `ssh`) messages would be "normal"?** @@ -652,16 +884,16 @@ Usually those messages will be displayed for the first instances to be configure However, an instance configuration step may indeed get stuck displaying those messages no end. It has happened to us under two conditions that we will call test conditions and normal conditions. -#### **Test conditions** -By test conditions we mean creating instances to test something (as opposed to creating instances for a workshop: normal conditions). In testing something, we have sometimes created a few instances and immediately after realised we forgot somethineg, corrected it, deleted the instances and created the instances again using the same instances names, and then got those messages no end. +###### **Test conditions** +By test conditions we mean creating instances to test something (as opposed to creating instances for a workshop: normal conditions). In testing something, we have sometimes created a few instances and immediately after realised we forgot somethineg, corrected it, deleted the instances and created the instances again using the **same** instance names, and then got those messages no end. We had to delete the instances, and create new ones with different names. We believe the problem is the too close subsequent requests to map the same domain names to different IP addresses. The AWS domain name system, Route 53, needs sometime to propagate the changes to its servers, etc. It has not been worth trying to solve this. They are only tests and we can use other instances names. -#### **Normal conditions** +###### **Normal conditions** Creating instances for a workshop is normal conditions. We have got those messages under normal conditions and the following has always worked for us: - **abort** the run of the script `csinstances_create.sh` by pressing `Ctrl-c` (the keys Ctrl and c simultaneously) a few times until you get back the prompt of your terminal. You will be aborting the script `aws_instances_configure.sh` first, the one that displays those messages, and then the script `csinstances_create.sh` which invoked `aws_instances_configure.sh` as the **last step** of creating instances.\ -Note that **the AWS resources of all instances have already been created** when you abort the Scripts while seeing the messages "Connection timed out" or "Connection refused". It is only the **instances configuration** phase that has not been completed. +Note that, when you abort the Scripts while seeing the messages "Connection timed out" or "Connection refused", **the AWS resources of all instances have already been created**. It is only the **instances configuration** phase that has not been completed. - **reboot** the instance that got stuck in the configuration phase --- the instance should be **Running** if all the previous steps were succesful (they should; check the messages displayed to your terminal). We usually **reboot** an instance in the AWS Console. Go to the **EC2 - Instances** page - check the box to the left of the instance name - click on **Instances state** drop-down menu at the top - select **Reboot Instance**. @@ -674,21 +906,24 @@ aws_instances_configure.sh courses/instances-management/inputs/instancesNames.tx ``` - this run will configure all the instances specified in the file *instancesNames.txt*. There is no problem if some instances were previously configured (before aborting the Scripts) and are configured again. - - alternatively, you may want to deal only with the instance on which the configuration step got stuck. In this case you need to create another "instancesNamesFile" containing the name of that instance only and run `aws_instances_configure.sh` with that file (see the previous section), for example: + - alternatively, you may want to deal only with the instance on which the configuration step got stuck. In this case you need to create another "instancesNamesFile" containing the name of that instance only and run `aws_instances_configure.sh` with that file (see the previous section [Unforseen instances management](#unforseen-instance-management)), for example: ``` {.bash filename="Code"} -aws_instances_configure.sh courses/instances-management/inputs/instancesNames20230104-configure-10.txt +aws_instances_configure.sh courses/instances-management/inputs/instancesNames20241028-configure-10.txt ``` - Alternatively, you may try first all of the steps above except **rebooting** the instance. It has worked for us. We believe the benefit of rebooting the instance is the cleaning/deleting of any previous `ssh` requests enqueued somewhere waiting to be served. -**The other scenario** where `Connection timed out /Connection refused` messages may be displayed is when an instance is to be accessed by the end user (after the instance has been configured and is running). In this case, it is the configuration of the server the end user is connected to gain access to the Internet. The server is blocking `ssh` communication, either from the end user `ssh` client to reach the instance or from the instance SSH server to reach the end user machine, or both. If you have a mobile phone handy, turn on the "Mobile Hotspot" WiFi, ask the end user to connect to it, and then to connect to the instance. If this works, the end user needs to ask the IT department responsible for the server to solve the issue. +##### **After configuration scenario** +The other scenario where **Connection timed out** or **Connection refused** messages may be displayed is when an instance is to be accessed by the end user (after the instance has been configured and is running). -### WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED +In this case, it is the configuration of the server the end user is connected to gain access to the Internet. The server is blocking `ssh` communication, either from the end user `ssh` client to reach the instance or from the instance SSH server to reach the end user machine, or both. If you have a mobile phone handy, turn on the "Mobile Hotspot" WiFi, ask the end user to connect to it, and then to connect to the instance. If this works, the end user needs to ask the IT department responsible for the server to solve the issue. + +### **WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED** The error behind this message has also to do with accessing and instance with `ssh`. It used to happen to us and happened to a workshop participant whose instance was created, deleted and created again with the same name. -The participant was sent the login key file the first time the instance was created. We then deleted the instance (instance321) in the file listing in section 7 due to not seeing any progress. The participant asked the instance to be created again. We created the instance again using the same name and send the participant the new login key file. When using `ssh` to login to the newly created instance, the participant got that message. +The participant was sent the login key file the first time the instance was created. We then deleted the instance (instance321 in the file listing in section [Unforseen instance management](#unforseen-instance-management) due to not seeing any progress). The participant asked the instance to be created again. We created the instance again using the same name and send the participant the new login key file. When using `ssh` to login to the newly created instance, the participant got that message. The solution was to tell the participant to run the command below before running `ssh` so that the instance host domain is removed from the participant's `~/.ssh/known_hosts` file. @@ -696,5 +931,4 @@ The solution was to tell the participant to run the command below before running ssh-keygen -f "$HOME/.ssh/known_hosts" -R instance321.cloud-span.aws.york.ac.uk ``` - -The error used to happen to us when developing the Scripts out of carrying out multiple tests using the same instances names. So we decided to add that `ssh-keygen` command within the script `aws_instances_configure.sh` just before invoking `ssh` --- see the screenshot above, fourth line from the top. +The error used to happen to us when developing the Scripts out of carrying out multiple tests using the same instances names. So we decided to add that `ssh-keygen` command to the script `aws_instances_configure.sh` just before invoking `ssh` --- see the screenshot above, eighth line from the top. diff --git a/docs/lesson02-managing-aws-instances/index.qmd b/docs/lesson02-managing-aws-instances/index.qmd index 065dec0..efa8a4b 100644 --- a/docs/lesson02-managing-aws-instances/index.qmd +++ b/docs/lesson02-managing-aws-instances/index.qmd @@ -1,58 +1,64 @@ --- title: Managing AWS Instances --- -Please read **[Workshops Organisation](/index.qmd#course-overview)** if you haven't done so. +Please read **[Workshops Organisation](/index.qmd#workshops-organisation)** if you haven't done so. The environments and configurations presented in the previous lesson, [Setting Up Your Cloud and Terminal Environments](/docs/lesson01-setting-work-envs/index.qmd), comprise a **base development environment** to create and manage AWS services with both the AWS Console and the AWS CLI (command line interface). If you are using an AWS **personal account** (that you created and configured as described in that lesson), you can create and manage **any service in any AWS region**, as your account was configured with the AdministratorAccess permissions policy which "*Provides full access to AWS services and resources*". -Towards using the Scripts to create and manage AWS instances, we will first need to configure **Internet access** for the instances. The instances configuration and the Scripts were designed for each instance to be accessed through a **domain name** and with the program `ssh` using an encrypted login key, as outlined below. Hence, we will create a **base domain name** from which the Scripts will create, for each instance, a **subdomain** name to identify and make each instance accessible with `ssh`. Creating a domain name in your AWS account will incur some cost, as low as US $5.00 per year or more depending on the *suffix /top-level domain* (TLD) that you choose for your base domain name. Examples of TLDs include: `.com`, `.net`, `.org`, `.link`, among many others (the cheapest in AWS is `.link`). If you already have a domain name in place, you can use that instead. Your account will also incur costs for any service you launch that is not included in the **AWS Free Tier** --- check the lesson [AWS Costs Explained](https://cloud-span.github.io/create-aws-instance-3-costs-explained/) from another Cloud-SPAN course, so you know your free limits. +Towards using the Scripts to create and manage AWS instances, we will first need to configure **Internet access** for the instances. The instances configuration and the Scripts were designed for each instance to be accessed with the program `ssh` using an encrypted login key and an **optional** *domain name*, as outlined below. If you decide on using domain names to access instances, you must then create a **base domain name** from which the Scripts will create a **subdomain** for each instance. Creating a base domain name in your AWS account will incur some cost, as low as US $5.00 per year or more depending on the *suffix /top-level domain* (TLD) that you choose for your base domain name. Examples of TLDs include: `.com`, `.net`, `.org`, `.link`, among many others (the cheapest in AWS is `.link`). If you already have a domain name in place, you can use that instead. Your account will also incur costs for any service you launch that is not included in the **AWS Free Tier** --- check the lesson [AWS Costs Explained](https://cloud-span.github.io/create-aws-instance-3-costs-explained/) from another Cloud-SPAN course, so you know your free limits. -If you are using an AWS **institutional account** (that was created and configured by the IT department in your institution), you will most likely not be directly responsible for the cost of using a domain name or any other service with your account. However, you may need to ask your IT department to configure a base domain name for you, or to follow some guidelines to specify the domain name. Your account may also have restrictions as to the **AWS region where you can create and manage AWS services**. For instance, the AWS institutionl account of the Cloud-SPAN project can only make use of services in the eu-west-1 Ireland region, and the Cloud-SPAN base domain name was suggested and configured by our IT department. If your institutional account is restricted to use an AWS region other than Ireland, you may need to ask your IT department to make a copy, to your AWS account, of the Amazon Machine Image (**AMI**) template used by the Scripts to create AWS instances. **Your IT department should be able to help you** with this and other matters; just let them know what you need to configure in your AWS account to use the Scripts, and whenever you come accross **Access Denied** or similar messages when using the Scripts. +If you are using an AWS **institutional account** (that was created and configured by the IT department in your institution), and you decide on using domain names, you will most likely not be directly responsible for the cost of using domain names or any other service with your account. However, you may need to ask your IT department to configure the base domain name for you, or to follow some guidelines to specify that domain name. Your account may also have restrictions as to the **AWS region where you can create and manage AWS services**. For instance, the AWS institutionl account of the Cloud-SPAN project can only make use of services in the eu-west-1 Ireland region, and the Cloud-SPAN base domain name was suggested and configured by our IT department. If your institutional account is restricted to use an AWS region other than Ireland, you may need to ask your IT department to make a copy, to your AWS account, of the Amazon Machine Image (**AMI**) template used by the Scripts to create AWS instances. **Your IT department should be able to help you** with this and other matters; just let them know what you need to configure in your AWS account to use the Scripts, and whenever you come accross **Access Denied** or similar messages when using the Scripts. # Overview -We use the Scripts to create and manage multiple AWS instances for training. When running a workshop, a number of instances is created as a copy of an AMI that is configured with ‘omics data and software analysis tools that are relevant to the workshop. Each student is granted exclusive access to one instance. +We use the Scripts to create and manage multiple AWS instances for training. When running a workshop, for each participant an instance is created as a copy of an AMI that is configured with ‘omics data and software analysis tools that are relevant to the workshop. -Each instance is created to be accessed through a **domain name** using `ssh` and an encrypted login key file. +Workshop participants login to the `csuser` account in their instance, using `ssh` with an encrypted login key file and **either** with a domain name (if a base domain name was configured and is specified to the Scripts as described later in the course) **or** with the IP address allocated to the instance by AWS. -For example, using the base domain name of the Cloud-SPAN project, `cloud-span.aws.york.ac.uk`, the (sub) domain name for an instance named `instance001` would be `instance001.cloud-span.aws.york.ac.uk`. +Examples. Suppose two instances have been created with the names `instance01` and `instance02` (which must be specified in a file that is passed to the Scripts as described later in the course). Each instance would be accessed as shown below (depending on whether a base domain name was specified to create the instances): -Using the base domain name of an AWS personal account, for example, `awsplaicloud.com`, the domain name for the same instance name would be `instance001.awsplaicloud.com`. +``` {.bash filename="Code"} +$ ssh -i login-key-instance01.pem csuser@instance01.cloud-span.aws.york.ac.uk +$ ssh -i login-key-instance02.pem csuser@instance02.cloud-span.aws.york.ac.uk -Once an instance is created, the end user will access the instance `csuser` account with `ssh` providing the name of the corresponding login key file as shown below. +$ ssh -i login-key-instance01.pem csuser@instance01.awsplaicloud.com +$ ssh -i login-key-instance02.pem csuser@instance02.awsplaicloud.com -Using the AWS Cloud-SPAN institutional account base domain name: -``` {.bash filename="Code"} -$ ssh -i login-key-instance001.pem csuser@instance001.cloud-span.aws.york.ac.uk ### -i stands for identity file +$ ssh -i login-key-instance01.pem csuser@10.134.43.226 +$ ssh -i login-key-instance02.pem csuser@43.226.10.134 ``` +The `ssh` parameter `-i` stands for identify file. -Using the personal account based domain name: -``` {.bash filename="Code"} -$ ssh -i login-key-instance001.pem csuser@instance001.awsplaicloud.com -``` +The first two lines correspond to the case where the base domain name specified was that of the Cloud-SPAN project institutional account: `cloud-span.aws.york.ac.uk`. -Each instance domain name is mapped to an IP address. Domain names, IP addresses, and login keys are created *automatically* on creating the corresponding instances, and deleted likewise when the corresponding instances are deleted. +The third and fourth lines correspond to the case where the base domain name specified was that of an AWS personal account, for example: `awsplaicloud.com`. + +The last two lines correspond to the case where **no** base domain name was specified. Note that in this case the instance names are still used to identify the login key files but cannot be used (as part of a domain name) to identify the corresponding instances. ### Episode 1: Configure Instances Internet Access Episode 1 will guide you to configure internet access for the instances you will create with the Scripts. This involves: -- creating a base domain name + +- creating a base domain name --- **optional** - creating a few access rules for the communication ports used by the `ssh` program -- selecting an AWS network to which the instances will be attached so that they can be reached from anywhere with `ssh` +- selecting an AWS network to which the instances will be attached so that they can be reached from anywhere with `ssh` --- **optional** but please read the overview of the episode + ### Episode 2: Instances Management Tasks Using the Scripts Episode 2 is the guide to using the Scripts to create and manage multiple instances for a course /workshop. The episode shows: + - how to configure the Scripts with the names of the instances to create and the AWS resources to use (base domain name, AMI template, etc.). - how to organise the Scripts configuration files for multiple courses. -- how to use /run the Scripts and manage unforseen instances management requests such as cancellations by workshop participants +- how to use /run the Scripts +- how to manage unforseen instances management requests such as cancellations or late registrations by workshop participants, and - some troubleshooting ### Episode 3: AMIs Management -Episode 3 is about managing Amazon Machine Images (AMIs). As AWS instances are copies of an AMI, you need to create a new AMI if the software or data used in a course change, but there are other reasons that may require creating a new AMI. The episode presents the management of AMIs that we have done as part of managing AWS instances with the Scripts. +Episode 3 is about managing Amazon Machine Images (AMIs). As AWS instances are copies of an AMI, you need to create a new AMI if the software or data used in a course change, but there are other reasons that may require creating a new AMI. The episode presents the management of AMIs that we have done as part of managing AWS instances with the Scripts, including how to create and customise a new AMI. ### Episode 4: The Scripts Design diff --git a/images/02-instances-management/05-results-of-running-csinstance_create.sh-outputs-dir.png b/images/02-instances-management/05-results-of-running-csinstance_create.sh-outputs-dir.png index 10f8117..e155fb3 100644 Binary files a/images/02-instances-management/05-results-of-running-csinstance_create.sh-outputs-dir.png and b/images/02-instances-management/05-results-of-running-csinstance_create.sh-outputs-dir.png differ diff --git a/images/02-instances-management/06-results-of-running-csinstance_create.sh-outputs-dir-files.png b/images/02-instances-management/06-results-of-running-csinstance_create.sh-outputs-dir-files.png index a8b9434..df97231 100644 Binary files a/images/02-instances-management/06-results-of-running-csinstance_create.sh-outputs-dir-files.png and b/images/02-instances-management/06-results-of-running-csinstance_create.sh-outputs-dir-files.png differ diff --git a/images/02-instances-management/07-results-of-running-csinstance_create.sh-instance01-id.png b/images/02-instances-management/07-results-of-running-csinstance_create.sh-instance01-id.png index cea7faf..7a51a8b 100644 Binary files a/images/02-instances-management/07-results-of-running-csinstance_create.sh-instance01-id.png and b/images/02-instances-management/07-results-of-running-csinstance_create.sh-instance01-id.png differ diff --git a/images/02-instances-management/08-no-domain-names-IP-addresses-results-files.png b/images/02-instances-management/08-no-domain-names-IP-addresses-results-files.png new file mode 100644 index 0000000..66c8f66 Binary files /dev/null and b/images/02-instances-management/08-no-domain-names-IP-addresses-results-files.png differ diff --git a/images/02-instances-management/11-terminal-listing-of-outpus-dir-after-stopping-starting-deleting-instances.png b/images/02-instances-management/11-terminal-listing-of-outpus-dir-after-stopping-starting-deleting-instances.png index 3aa06e8..1ab8fe4 100644 Binary files a/images/02-instances-management/11-terminal-listing-of-outpus-dir-after-stopping-starting-deleting-instances.png and b/images/02-instances-management/11-terminal-listing-of-outpus-dir-after-stopping-starting-deleting-instances.png differ diff --git a/images/02-instances-management/12-results-of-running-csinstance_create.sh-configuring-instance2.png b/images/02-instances-management/12-results-of-running-csinstance_create.sh-configuring-instance2.png index 62ebead..facf42c 100644 Binary files a/images/02-instances-management/12-results-of-running-csinstance_create.sh-configuring-instance2.png and b/images/02-instances-management/12-results-of-running-csinstance_create.sh-configuring-instance2.png differ diff --git a/images/02-instances-management/13-results-of-running-csinstance_create.sh-AGAIN-aborted.png b/images/02-instances-management/13-results-of-running-csinstance_create.sh-AGAIN-aborted.png index e7d601a..f99b9fe 100644 Binary files a/images/02-instances-management/13-results-of-running-csinstance_create.sh-AGAIN-aborted.png and b/images/02-instances-management/13-results-of-running-csinstance_create.sh-AGAIN-aborted.png differ diff --git a/images/02-instances-management/fig01-scripts-run-validating-config-files.png b/images/02-instances-management/fig01-scripts-run-validating-config-files.png new file mode 100644 index 0000000..dcbf1ef Binary files /dev/null and b/images/02-instances-management/fig01-scripts-run-validating-config-files.png differ diff --git a/images/02-instances-management/fig02-scripts-run-creating-instances.png b/images/02-instances-management/fig02-scripts-run-creating-instances.png new file mode 100644 index 0000000..bde6dc6 Binary files /dev/null and b/images/02-instances-management/fig02-scripts-run-creating-instances.png differ diff --git a/images/02-instances-management/fig03-scripts-aws-infrastructure-n-users.png b/images/02-instances-management/fig03-scripts-aws-infrastructure-n-users.png new file mode 100644 index 0000000..50bcf3b Binary files /dev/null and b/images/02-instances-management/fig03-scripts-aws-infrastructure-n-users.png differ diff --git a/images/02-instances-management/fig03-scripts-run-configuring-instances.png b/images/02-instances-management/fig03-scripts-run-configuring-instances.png new file mode 100644 index 0000000..ef7430b Binary files /dev/null and b/images/02-instances-management/fig03-scripts-run-configuring-instances.png differ diff --git a/images/config-linux-env/04-aws-cloudshell-ready-screen.png b/images/config-linux-env/04-aws-cloudshell-ready-screen.png index 76af037..f2aa13f 100644 Binary files a/images/config-linux-env/04-aws-cloudshell-ready-screen.png and b/images/config-linux-env/04-aws-cloudshell-ready-screen.png differ diff --git a/images/config-linux-env/05-aws-cloudshell-cloning-the-scripts.png b/images/config-linux-env/05-aws-cloudshell-cloning-the-scripts.png index b731999..2d98e67 100644 Binary files a/images/config-linux-env/05-aws-cloudshell-cloning-the-scripts.png and b/images/config-linux-env/05-aws-cloudshell-cloning-the-scripts.png differ diff --git a/images/config-linux-env/06-aws-cloudshell-installing-the-scripts-n-running-csinstances_create.png b/images/config-linux-env/06-aws-cloudshell-installing-the-scripts-n-running-csinstances_create.png index 43ca670..248d525 100644 Binary files a/images/config-linux-env/06-aws-cloudshell-installing-the-scripts-n-running-csinstances_create.png and b/images/config-linux-env/06-aws-cloudshell-installing-the-scripts-n-running-csinstances_create.png differ diff --git a/images/config-linux-env/07-aws-cloudshell-actions-menu.png b/images/config-linux-env/07-aws-cloudshell-actions-menu.png index a8c5a62..d5ae0f3 100644 Binary files a/images/config-linux-env/07-aws-cloudshell-actions-menu.png and b/images/config-linux-env/07-aws-cloudshell-actions-menu.png differ diff --git a/images/config-linux-env/08-linux-terminal-running-csinstances_create-ScriptsV2.png b/images/config-linux-env/08-linux-terminal-running-csinstances_create-ScriptsV2.png new file mode 100644 index 0000000..992fb0f Binary files /dev/null and b/images/config-linux-env/08-linux-terminal-running-csinstances_create-ScriptsV2.png differ diff --git a/images/config-linux-env/08-linux-terminal-running-csinstances_create.png b/images/config-linux-env/08-linux-terminal-running-csinstances_create.png deleted file mode 100644 index fd9e647..0000000 Binary files a/images/config-linux-env/08-linux-terminal-running-csinstances_create.png and /dev/null differ