Merge pull request #226 from aerospike/v7.4.0

V7.4.0

CHANGELOG.md

@@ -1,3 +1,16 @@
+#### 7.4.0
+* FEATURE: Web UI.
+* FIX: GCP: Many commands would fail during template creation, making parallel use imposible. Fixed.
+* FIX: GCP: Delete `arm` templates was not working at all.
+* FIX: The `net list` command does not work when client has same name as server.
+* FIX: The `net loss-delay` feature would fail to activate a python environment.
+* ENHANCEMENT: Support all ubuntu 18+ and centos 7+ builds with `net loss-delay` feature.
+* ENHANCEMENT: When `--on-destination` is selected in `net loss-delay`, set `--src-network` instead of `--network`.
+* ENHANCEMENT: The `net loss-delay` feature now supports specifying ports.
+* ENHANCEMENT: The `logs get` command will append original filename as suffix and will ask if files would be overwritten unless `-f` is specified.
+* ENHANCEMENT: For centos stream 8/9 installs, there is no more need to re-enable repos and sync distros.
+* ENHANCEMENT: All inventory instance listings in cloud will now show instance type in the last field.
+
 #### 7.3.0
 * Volume command supports mounts on centos/amazon.
 * Version check logic - if user is using a dev build and a final build is out, inform user of available upgrade.

LATEST.md

@@ -1,42 +1,16 @@
-# [v7.3.0](https://github.com/aerospike/aerolab/releases/tag/7.3.0)
+# [v7.4.0](https://github.com/aerospike/aerolab/releases/tag/7.4.0)
 
 _Release Date: January 2, 2024_
 
 **Release Notes:**
-* Volume command supports mounts on centos/amazon.
-* Version check logic - if user is using a dev build and a final build is out, inform user of available upgrade.
-* Allow choosing renderer mode for inventory listing to be normal, TSV, CSV, Markdown or HTML.
-* Tag expiry systems to allow for version discovery and automated updating.
-* Add `aerolab upgrade [--edge]` command to allow upgrading to latest stable (or latest pre-release).
-* Windows: catch the use of command prompt and warn against it.
-* Improvement: replace the `mesh/multicast` fix system with `aeroconf` parser.
-* Improvement: in `aerolab conf namespace-memory`, adjust `data-size` instead if using aerospike `v7+`, and only if required. Otherwise `noop`.
-* Allow adding of `aerolab` to a server/client instance on-demand.
-* AGI commands support centos/amazon.
-* AGI: support v7 of aerospike.
-* AGI: support arm editions of centos-based operating systems.
-* AGI: add `--no-dim-filesize` option to specify data storage file size for non-data-in-memory namespaces.
-* AGI: override tools package by default by the latest tools. Allow `--no-tools-override` to disable.
-* AGI: All stack items utilize `v7` aerospike client library.
-* AGI: Feature: Monitor agi instance states and react accordingly - sizing or cycling from spot to on-demand types.
-* Docker: Improvement: attempt to auto-adjust limit of open files.
-* Docker: when exposing multiple ports in a continuous order, only use first exposed port for the service port.
-* GCP: The `cluster add firewall` command should issue `op.Wait` after all operations have been queued.
-* GCP AGI spot instance support.
-* GCP Volumes support (using standard `pd-ssd`, and requiring size, as there is no `EFS` equivalent).
-* AWS: Always encrypt EBS volumes.
-* Fix: `conf generate` command for v7 aerospike is missing an argument `data-size` where needed.
-* Fix: add support for `time.Duration` to `config defaults` command.
-* Fix: `config defaults` should be able to set defaults for any parameter, even if it is for a backend not currently in use.
-* Fix: aws security groups will now also open ICMP traffic in the rules.
-* Fix: GCP: for some instances, `TERMINATE` scheduling policy must be set.
-* Fix: Docker show IPs for containers which do not have exposed ports.
-* Fix: AMS client fix regression: `asbench` dashboard installation.
-* Fix: Auto-create temporary directory as needed, if it is specified.
-* Fix: GCP: Handle firewall rules which use an IP instead of CIDR.
-* Fix: New exporter name changes cause 1.15.0 to not install. This uses the new names.
-* Fix: `lsblk` now shows sizes as float not int.
-* Fix: GCP: upgrade and improve instance type listing.
-* Fix: AWS: multiple APIs when user specifies AWS backend without choosing a custom region.
-* GCP: instance-type listing now includes spot instance support.
-* Aerospike Client Versions: `data insert`, `data delete` support for `v7` aerospike client.
+* FEATURE: Web UI.
+* FIX: GCP: Many commands would fail during template creation, making parallel use imposible. Fixed.
+* FIX: GCP: Delete `arm` templates was not working at all.
+* FIX: The `net list` command does not work when client has same name as server.
+* FIX: The `net loss-delay` feature would fail to activate a python environment.
+* ENHANCEMENT: Support all ubuntu 18+ and centos 7+ builds with `net loss-delay` feature.
+* ENHANCEMENT: When `--on-destination` is selected in `net loss-delay`, set `--src-network` instead of `--network`.
+* ENHANCEMENT: The `net loss-delay` feature now supports specifying ports.
+* ENHANCEMENT: The `logs get` command will append original filename as suffix and will ask if files would be overwritten unless `-f` is specified.
+* ENHANCEMENT: For centos stream 8/9 installs, there is no more need to re-enable repos and sync distros.
+* ENHANCEMENT: All inventory instance listings in cloud will now show instance type in the last field.

RELEASE.md

@@ -2,16 +2,16 @@
 
 If using AWS or GCP backends, extra steps are required to migrate the firewalls from AeroLab version 6.0 or below to a new system.
 
-[Follow this manual for upgrade instructions](https://github.com/aerospike/aerolab/blob/v7.3.0/docs/upgrade-to-610.md)
+[Follow this manual for upgrade instructions](https://github.com/aerospike/aerolab/blob/v7.4.0/docs/upgrade-to-610.md)
 
 NOTES:
 * AeroLab 7.0.0 implements an instance expiry system. By default your instances will terminate after 30 hours. To modify this behaviour, create clusters with `--aws-expires TIME` or `--gcp-expires`. For example `--aws-expires 50h`. To disable expiry, set to `0`.
 * AeroLab 7.2.0 adds AWS EFS Volume expiries. If using the expiry system in AWS, reinstall using `aerolab config aws expiry-remove && aerolab config aws expiry-install`.
 
 ### Documentation and changelog
-See [the documentation](https://github.com/aerospike/aerolab/blob/v7.3.0/README.md) for full installation and usage instructions.
+See [the documentation](https://github.com/aerospike/aerolab/blob/v7.4.0/README.md) for full installation and usage instructions.
 
-[Changelog](https://github.com/aerospike/aerolab/blob/v7.3.0/CHANGELOG.md#7.3.0)
+[Changelog](https://github.com/aerospike/aerolab/blob/v7.4.0/CHANGELOG.md#7.4.0)
 
 ### Download aerolab from Assets below
 

VERSION.md

@@ -1 +1 @@
-7.3.0
+7.4.0

src/backendDocker.go

@@ -385,7 +385,17 @@ func (d *backendDocker) ClusterList() ([]string, error) {
 }
 
 func (d *backendDocker) NodeListInCluster(name string) ([]int, error) {
-	out, err := exec.Command("docker", "container", "list", "-a", "--format", "{{json .Names}}").CombinedOutput()
+	return d.nodeListInClusterDo(name, false)
+}
+
+func (d *backendDocker) nodeListInClusterDo(name string, onlyRunning bool) ([]int, error) {
+	var out []byte
+	var err error
+	if onlyRunning {
+		out, err = exec.Command("docker", "container", "list", "--format", "{{json .Names}}").CombinedOutput()
+	} else {
+		out, err = exec.Command("docker", "container", "list", "-a", "--format", "{{json .Names}}").CombinedOutput()
+	}
 	if err != nil {
 		return nil, err
 	}
@@ -938,7 +948,7 @@ func (d *backendDocker) GetNodeIpMap(name string, internalIPs bool) (map[int]str
 	if !inslice.HasString(clusters, name) {
 		return nil, errors.New("cluster not found")
 	}
-	nodes, err := d.NodeListInCluster(name)
+	nodes, err := d.nodeListInClusterDo(name, true)
 	if err != nil {
 		return nil, err
 	}

src/backendGcp.go

@@ -1801,6 +1801,9 @@ func (d *backendGcp) GetNodeIpMap(name string, internalIPs bool) (map[int]string
 		instances := pair.Value.Instances
 		if len(instances) > 0 {
 			for _, instance := range instances {
+				if strings.HasPrefix(*instance.Name, "aerolab4-template-") {
+					continue
+				}
 				if instance.Labels[gcpTagUsedBy] == gcpTagUsedByValue {
 					if instance.Labels[gcpTagClusterName] == name {
 						nodeNo, err := strconv.Atoi(instance.Labels[gcpTagNodeNumber])
@@ -2240,7 +2243,7 @@ func (d *backendGcp) TemplateDestroy(v backendVersion) error {
 		}
 		if image.Labels[gcpTagUsedBy] == gcpTagUsedByValue {
 			isArm := false
-			if strings.Contains(*image.Architecture, "arm") || strings.Contains(*image.Architecture, "aarch") {
+			if strings.Contains(strings.ToLower(*image.Architecture), "arm") || strings.Contains(strings.ToLower(*image.Architecture), "aarch") {
 				isArm = true
 			}
 			if (image.Labels[gcpServerTagOperatingSystem] == v.distroName || v.distroName == "all") && (image.Labels[gcpServerTagOSVersion] == gcpResourceName(v.distroVersion) || v.distroVersion == "all") && (image.Labels[gcpServerTagAerospikeVersion] == gcpResourceName(v.aerospikeVersion) || v.aerospikeVersion == "all") && isArm == v.isArm {

src/cmdInventoryList.go

@@ -421,9 +421,9 @@ func (c *inventoryListCmd) run(showClusters bool, showClients bool, showTemplate
 		t.ResetRows()
 		t.ResetFooters()
 		if a.opts.Config.Backend.Type == "gcp" {
-			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "ExpiresIn", "State", "PublicIP", "PrivateIP", "Owner", "AsdVer", "RunningCost", "Firewalls", "Arch", "Distro", "DistroVer", "Zone", "InstanceID"})
+			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "ExpiresIn", "State", "PublicIP", "PrivateIP", "Owner", "AsdVer", "RunningCost", "Firewalls", "Arch", "Distro", "DistroVer", "Zone", "InstanceID", "InstanceType"})
 		} else if a.opts.Config.Backend.Type == "aws" {
-			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "ExpiresIn", "State", "PublicIP", "PrivateIP", "Owner", "AsdVer", "RunningCost", "Firewalls", "Arch", "Distro", "DistroVer", "Region", "InstanceID"})
+			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "ExpiresIn", "State", "PublicIP", "PrivateIP", "Owner", "AsdVer", "RunningCost", "Firewalls", "Arch", "Distro", "DistroVer", "Region", "InstanceID", "InstanceType"})
 		} else {
 			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "State", "PublicIP", "PrivateIP", "ExposedPort", "Owner", "AsdVer", "Arch", "Distro", "DistroVer", "InstanceID", "ImageID"})
 		}
@@ -474,6 +474,9 @@ func (c *inventoryListCmd) run(showClusters bool, showClients bool, showTemplate
 			vv = append(vv, v.InstanceId)
 			if a.opts.Config.Backend.Type == "docker" {
 				vv = append(vv, v.ImageId)
+			} else {
+				itype := strings.Split(v.InstanceType, "/")
+				vv = append(vv, itype[len(itype)-1])
 			}
 			t.AppendRow(vv)
 		}
@@ -492,9 +495,9 @@ func (c *inventoryListCmd) run(showClusters bool, showClients bool, showTemplate
 		t.ResetRows()
 		t.ResetFooters()
 		if a.opts.Config.Backend.Type == "gcp" {
-			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "ExpiresIn", "State", "PublicIP", "PrivateIP", "ClientType", "AccessURL", "AccessPort", "Owner", "AsdVer", "RunningCost", "Firewalls", "Arch", "Distro", "DistroVer", "Zone", "InstanceID"})
+			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "ExpiresIn", "State", "PublicIP", "PrivateIP", "ClientType", "AccessURL", "AccessPort", "Owner", "AsdVer", "RunningCost", "Firewalls", "Arch", "Distro", "DistroVer", "Zone", "InstanceID", "InstanceType"})
 		} else if a.opts.Config.Backend.Type == "aws" {
-			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "ExpiresIn", "State", "PublicIP", "PrivateIP", "ClientType", "AccessURL", "AccessPort", "Owner", "AsdVer", "RunningCost", "Firewalls", "Arch", "Distro", "DistroVer", "Region", "InstanceID"})
+			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "ExpiresIn", "State", "PublicIP", "PrivateIP", "ClientType", "AccessURL", "AccessPort", "Owner", "AsdVer", "RunningCost", "Firewalls", "Arch", "Distro", "DistroVer", "Region", "InstanceID", "InstanceType"})
 		} else {
 			t.AppendHeader(table.Row{"ClusterName", "NodeNo", "State", "PublicIP", "PrivateIP", "ClientType", "AccessURL", "AccessPort", "Owner", "AsdVer", "Arch", "Distro", "DistroVer", "InstanceID", "ImageID"})
 		}
@@ -539,6 +542,9 @@ func (c *inventoryListCmd) run(showClusters bool, showClients bool, showTemplate
 			vv = append(vv, v.InstanceId)
 			if a.opts.Config.Backend.Type == "docker" {
 				vv = append(vv, v.ImageId)
+			} else {
+				itype := strings.Split(v.InstanceType, "/")
+				vv = append(vv, itype[len(itype)-1])
 			}
 			t.AppendRow(vv)
 		}
@@ -662,9 +668,9 @@ func (c *inventoryListCmd) run(showClusters bool, showClients bool, showTemplate
 			t.ResetFooters()
 			if showOther&inventoryShowAGIStatus > 0 {
 				if a.opts.Config.Backend.Type == "gcp" {
-					t.AppendHeader(table.Row{"Name", "State", "Status", "ExpiresIn", "VolOwner", "Owner", "Access URL", "AGILabel", "VolSize", "VolExpires", "RunningCost", "PublicIP", "PrivateIP", "Firewalls", "Zone", "InstanceID", "ExpiryTs", "VolExpiryTs"})
+					t.AppendHeader(table.Row{"Name", "State", "Status", "ExpiresIn", "VolOwner", "Owner", "Access URL", "AGILabel", "VolSize", "VolExpires", "RunningCost", "PublicIP", "PrivateIP", "Firewalls", "Zone", "InstanceID", "ExpiryTs", "VolExpiryTs", "InstanceType"})
 				} else if a.opts.Config.Backend.Type == "aws" {
-					t.AppendHeader(table.Row{"Name", "State", "Status", "ExpiresIn", "VolOwner", "Owner", "Access URL", "AGILabel", "VolSize", "VolExpires", "RunningCost", "PublicIP", "PrivateIP", "Firewalls", "Region", "VolID", "InstanceID", "ExpiryTs", "VolExpiryTs"})
+					t.AppendHeader(table.Row{"Name", "State", "Status", "ExpiresIn", "VolOwner", "Owner", "Access URL", "AGILabel", "VolSize", "VolExpires", "RunningCost", "PublicIP", "PrivateIP", "Firewalls", "Region", "VolID", "InstanceID", "ExpiryTs", "VolExpiryTs", "InstanceType"})
 				} else {
 					t.AppendHeader(table.Row{"Name", "State", "Status", "Owner", "Access URL", "AGILabel", "PublicIP", "PrivateIP", "InstanceID", "ImageID"})
 				}
@@ -848,6 +854,8 @@ func (c *inventoryListCmd) run(showClusters bool, showClients bool, showTemplate
 						} else {
 							vv = append(vv, " ")
 						}
+						itype := strings.Split(v.InstanceType, "/")
+						vv = append(vv, itype[len(itype)-1])
 					}
 					t.AppendRow(vv)
 				}
@@ -898,9 +906,9 @@ func (c *inventoryListCmd) run(showClusters bool, showClients bool, showTemplate
 				}
 			} else {
 				if a.opts.Config.Backend.Type == "gcp" {
-					t.AppendHeader(table.Row{"Name", "State", "ExpiresIn", "Owner", "Access URL", "AGILabel", "RunningCost", "PublicIP", "PrivateIP", "Firewalls", "Zone", "InstanceID", "ExpiryTs"})
+					t.AppendHeader(table.Row{"Name", "State", "ExpiresIn", "Owner", "Access URL", "AGILabel", "RunningCost", "PublicIP", "PrivateIP", "Firewalls", "Zone", "InstanceID", "ExpiryTs", "InstanceType"})
 				} else if a.opts.Config.Backend.Type == "aws" {
-					t.AppendHeader(table.Row{"Name", "State", "ExpiresIn", "Owner", "Access URL", "AGILabel", "RunningCost", "PublicIP", "PrivateIP", "Firewalls", "Region", "InstanceID", "ExpiryTs"})
+					t.AppendHeader(table.Row{"Name", "State", "ExpiresIn", "Owner", "Access URL", "AGILabel", "RunningCost", "PublicIP", "PrivateIP", "Firewalls", "Region", "InstanceID", "ExpiryTs", "InstanceType"})
 				} else {
 					t.AppendHeader(table.Row{"Name", "State", "Owner", "Access URL", "AGILabel", "PublicIP", "PrivateIP", "InstanceID", "ImageID"})
 				}
@@ -959,6 +967,8 @@ func (c *inventoryListCmd) run(showClusters bool, showClients bool, showTemplate
 					} else {
 						expirationTime, _ := time.Parse(time.RFC3339, v.Expires)
 						vv = append(vv, expirationTime.Unix())
+						itype := strings.Split(v.InstanceType, "/")
+						vv = append(vv, itype[len(itype)-1])
 					}
 					t.AppendRow(vv)
 				}

src/cmdLogsGet.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"bufio"
 	"errors"
 	"fmt"
 	"log"
@@ -20,6 +21,7 @@ type logsGetCmd struct {
 	Journal     bool            `short:"j" long:"journal" description:"Attempt to get logs from journald instead of log files"`
 	LogLocation string          `short:"p" long:"path" description:"Aerospike log file path" default:"/var/log/aerospike.log"`
 	Destination flags.Filename  `short:"d" long:"destination" description:"Destination directory (will be created if doesn't exist)" default:"./logs/"`
+	Force       bool            `short:"f" long:"force" description:"set to not be asked whether to override existing files"`
 	parallelThreadsCmd
 	Help helpCmd `command:"help" subcommands-optional:"true" description:"Print help"`
 }
@@ -74,6 +76,36 @@ func (c *logsGetCmd) Execute(args []string) error {
 		if err != nil {
 			return err
 		}
+	} else if !c.Force {
+		entries, _ := os.ReadDir(string(c.Destination))
+		_, logf := path.Split(c.LogLocation)
+		ask := false
+		for _, ee := range entries {
+			if strings.HasPrefix(ee.Name(), string(c.ClusterName)+"-") && strings.HasSuffix(ee.Name(), "."+strings.TrimLeft(logf, ".")) {
+				ask = true
+				break
+			}
+		}
+		if ask {
+			for {
+				reader := bufio.NewReader(os.Stdin)
+				fmt.Print("Directory exists and existing files will be overwritten, continue download (y/n)? ")
+
+				yesno, err := reader.ReadString('\n')
+				if err != nil {
+					logExit(err)
+				}
+
+				yesno = strings.ToLower(strings.TrimSpace(yesno))
+
+				if yesno == "y" || yesno == "yes" {
+					break
+				} else if yesno == "n" || yesno == "no" {
+					fmt.Println("Aborting")
+					return nil
+				}
+			}
+		}
 	}
 
 	c.Destination = flags.Filename(path.Join(string(c.Destination), string(c.ClusterName)))
@@ -117,6 +149,12 @@ func (c *logsGetCmd) getParallel(node int, parallel chan int, wait *sync.WaitGro
 
 func (c *logsGetCmd) get(node int) error {
 	fn := string(c.Destination) + "-" + strconv.Itoa(node)
+	if c.Journal {
+		fn = fn + ".journald.log"
+	} else {
+		_, logf := path.Split(c.LogLocation)
+		fn = fn + "." + strings.TrimLeft(logf, ".")
+	}
 	f, err := os.OpenFile(fn, os.O_CREATE|os.O_TRUNC|os.O_RDWR, 0644)
 	if err != nil {
 		return err