Updates tools to include mapmerge hooks and merge drivers, plus more (P…

…aradiseSS13#18034)

* bumps tools to modern versions, adds DMM merge driver hooks for automatic map merging

* mapping guidelines update

* Update .github/CONTRIBUTING.md

Co-authored-by: AffectedArc07 <[email protected]>

* aa review

* these somehow got ignored? force adding

* Apply suggestions from code review

Co-authored-by: AffectedArc07 <[email protected]>

.github/CONTRIBUTING.md

@@ -505,15 +505,14 @@ in the SQL/updates folder.
 * For map edit PRs, we do not accept 'change for the sake of change' remaps, unless you have very good reasoning to do so. Maintainers reserve the right to close your PR if we disagree with your reasoning.
 
 * Map Merge
-  * The following guideline for map merging applies to people who are **NOT** using StrongDMM, please see the StrongDMM section if you are.
-    * You **MUST** run Map Merge prior to opening your PR when updating existing maps to minimize the change differences (even when using third party mapping programs such as FastDMM.)
-    * Failure to run Map Merge on a map after using third party mapping programs (such as FastDMM) greatly increases the risk of the map's key dictionary becoming corrupted by future edits after running map merge. Resolving the corruption issue involves rebuilding the map's key dictionary;
+  * The following guideline for map merging applies to **ALL** mapping contributers.
+    * Before committing a map change, you **MUST** run mapmerge2 to normalise your changes. You can do this manually before every commit with `"\tools\mapmerge2\Run Before Committing.bat"` or automatically by installing the hooks at `"\tools\hooks\Install.bat"`.
+    * Failure to run Map Merge on a map after editing greatly increases the risk of the map's key dictionary becoming corrupted by future edits after running map merge. Resolving the corruption issue involves rebuilding the map's key dictionary;
 
 * StrongDMM
-  * When using StrongDMM, the following options **MUST** be enabled to avoid file bloat. They can be found under `File > Preferences > Save Options` in SDMM.
-    * Map save format: This **MUST** be set to **TGM** if you do not want to run Map Merge. Enabling this setting means SDMM will automatically map merge, letting you skip manual merging.
+  * When using StrongDMM, the following options should be enabled to avoid file bloat. They can be found under `File > Preferences` in SDMM2.
     * Sanitize Variables - Removes variables that are declared on the map, but are the same as default. (For example: A standard floor turf that has `dir = 2` declared on the map will have that variable deleted as it is redundant.)
-    * Clean Unused Keys - Removes content tile keys that are no longer used on the map, usually leftover keys from deletions or edits.
+    * Save format - Either `Initial` or `TGM`, never `DM`.
 
 * Variable Editing (Var-edits)
   * While var-editing an item within the editor is perfectly fine, it is preferred that when you are changing the base behavior of an item (how it functions) that you make a new subtype of that item within the code, especially if you plan to use the item in multiple locations on the same map, or across multiple maps. This makes it easier to make corrections as needed to all instances of the item at one time as opposed to having to find each instance of it and change them all individually.

.gitignore

@@ -35,3 +35,20 @@ stddef.dm
 
 # ignore midi2piano build cache
 /tools/midi2piano/midi2piano/obj/*
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# ignore python cache
+.cache
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# pyenv
+.python-version

_build_dependencies.sh

@@ -7,3 +7,5 @@ export NODE_VERSION=12
 export BYOND_MAJOR=514
 # Byond Minor
 export BYOND_MINOR=1566
+# Python version for mapmerge and other tools
+export PYTHON_VERSION=3.7.9

tools/UpdatePaths/Update_Paths.bat

@@ -0,0 +1,3 @@
+@echo off
+call "%~dp0\..\bootstrap\python" -m UpdatePaths %*
+pause

tools/mapmerge2/update_paths.py → tools/UpdatePaths/__main__.py

@@ -1,17 +1,19 @@
 # A script and syntax for applying path updates to maps.
-import re
-import os
-import argparse
-import frontend
-from dmm import *
+import re, os, sys, argparse
+from mapmerge2 import frontend
+from mapmerge2.dmm import *
 
 desc = """
 Update dmm files given update file/string.
 Replacement syntax example:
-    /turf/open/floor/plasteel/warningline : /obj/effect/turf_decal {dir = @OLD ;tag = @SKIP;icon_state = @SKIP}
-    /turf/open/floor/plasteel/warningline : /obj/effect/turf_decal {@OLD} , /obj/thing {icon_state = @OLD:name; name = "meme"}
-    /turf/open/floor/plasteel/warningline{dir=2} : /obj/thing
+    /turf/open/floor/iron/warningline : /obj/effect/turf_decal {dir = @OLD ;tag = @SKIP;icon_state = @SKIP}
+    /turf/open/floor/iron/warningline : /obj/effect/turf_decal {@OLD} , /obj/thing {icon_state = @OLD:name; name = "meme"}
+    /turf/open/floor/iron/warningline{dir=2} : /obj/thing
+    /obj/effect/landmark/start/virologist : @DELETE
+Syntax for subtypes also exist, to update a path's type but maintain subtypes:
+    /obj/structure/closet/crate/@SUBTYPES : /obj/structure/new_box/@SUBTYPES {@OLD}
 New paths properties:
+    @DELETE - if used as new path name the old path will be deleted
     @OLD - if used as property name copies all modified properties from original path to this one
     property = @SKIP - will not copy this property through when global @OLD is used.
     property = @OLD - will copy this modified property from original object even if global @OLD is not used
@@ -23,14 +25,14 @@
 """
 
 default_map_directory = "../../_maps"
-replacement_re = re.compile('\s*([^{]*)\s*(\{(.*)\})?')
+replacement_re = re.compile(r'\s*(?P<path>[^{]*)\s*(\{(?P<props>.*)\})?')
 
 #urgent todo: replace with actual parser, this is slow as janitor in crit
-split_re = re.compile('((?:[A-Za-z0-9_\-$]+)\s*=\s*(?:"(?:.+?)"|[^";]*)|@OLD)')
+split_re = re.compile(r'((?:[A-Za-z0-9_\-$]+)\s*=\s*(?:"(?:.+?)"|[^";][^;]*)|@OLD);?')
 
 
 def props_to_string(props):
-    return "{{{0}}}".format(";".join([k+" = "+props[k] for k in props]))
+    return "{{{}}}".format(";".join([f"{k} = {v}" for k, v in props.items()]))
 
 
 def string_to_props(propstring, verbose = False):
@@ -48,8 +50,8 @@ def string_to_props(propstring, verbose = False):
 def parse_rep_string(replacement_string, verbose = False):
     # translates /blah/blah {meme = "test",} into path,prop dictionary tuple
     match = re.match(replacement_re, replacement_string)
-    path = match.group(1)
-    props = match.group(3)
+    path = match['path']
+    props = match['props']
     if props:
         prop_dict = string_to_props(props, verbose)
     else:
@@ -65,9 +67,18 @@ def update_path(dmm_data, replacement_string, verbose=False):
         new_path, new_path_props = parse_rep_string(replacement_def, verbose)
         new_paths.append((new_path, new_path_props))
 
+    subtypes = ""
+    if old_path.endswith("/@SUBTYPES"):
+        old_path = old_path[:-len("/@SUBTYPES")]
+        if verbose:
+            print("Looking for subtypes of", old_path)
+        subtypes = r"(?:/\w+)*"
+
+    replacement_pattern = re.compile(rf"(?P<path>{re.escape(old_path)}(?P<subtype>{subtypes}))\s*(:?{{(?P<props>.*)}})?$")
+
     def replace_def(match):
-        if match.group(2):
-            old_props = string_to_props(match.group(2), verbose)
+        if match['props']:
+            old_props = string_to_props(match['props'], verbose)
         else:
             old_props = dict()
         for filter_prop in old_path_props:
@@ -83,7 +94,18 @@ def replace_def(match):
             print("Found match : {0}".format(match.group(0)))
         out_paths = []
         for new_path, new_props in new_paths:
-            out = new_path
+            if new_path == "@OLD":
+                out = match.group('path')
+            elif new_path == "@DELETE":
+                if verbose:
+                    print("Deleting match : {0}".format(match.group(0)))
+                return [None]
+            elif new_path.endswith("/@SUBTYPES"):
+                path_start = new_path[:-len("/@SUBTYPES")]
+                out = path_start + match.group('subtype')
+            else:
+                out = new_path
+
             out_props = dict()
             for prop_name, prop_value in new_props.items():
                 if prop_name == "@OLD":
@@ -106,29 +128,33 @@ def replace_def(match):
         return out_paths
 
     def get_result(element):
-        p = re.compile("{0}\s*({{(.*)}})?$".format(re.escape(old_path)))
-        match = p.match(element)
+        match = replacement_pattern.match(element)
         if match:
-            return replace_def(match) # = re.sub(p,replace_def,element)
+            return replace_def(match)
         else:
             return [element]
 
     bad_keys = {}
+    modified_keys = []
     keys = list(dmm_data.dictionary.keys())
     for definition_key in keys:
         def_value = dmm_data.dictionary[definition_key]
-        new_value = tuple(y for x in def_value for y in get_result(x))
+        new_value = tuple(y for x in def_value for y in get_result(x) if y != None)
         if new_value != def_value:
             dmm_data.overwrite_key(definition_key, new_value, bad_keys)
+            modified_keys.append(definition_key)
     dmm_data.reassign_bad_keys(bad_keys)
+    return modified_keys
 
 
 def update_map(map_filepath, updates, verbose=False):
     print("Updating: {0}".format(map_filepath))
     dmm_data = DMM.from_file(map_filepath)
+    modified_keys = []
     for update_string in updates:
-        update_path(dmm_data, update_string, verbose)
-    dmm_data.to_file(map_filepath, True)
+        modified_keys.extend(update_path(dmm_data, update_string, verbose))
+    dmm_data.remove_unused_keys(modified_keys)
+    dmm_data.to_file(map_filepath)
 
 
 def update_all_maps(map_directory, updates, verbose=False):
@@ -141,10 +167,12 @@ def update_all_maps(map_directory, updates, verbose=False):
 
 def main(args):
     if args.inline:
+        print("Using replacement:", args.update_source)
         updates = [args.update_source]
     else:
         with open(args.update_source) as f:
             updates = [line for line in f if line and not line.startswith("#") and not line.isspace()]
+        print(f"Using {len(updates)} replacements from file:", args.update_source)
 
     if args.map:
         update_map(args.map, updates, verbose=args.verbose)
@@ -154,7 +182,11 @@ def main(args):
 
 
 if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description=desc, formatter_class=argparse.RawTextHelpFormatter)
+    prog = __spec__.name.replace('.__main__', '')
+    if os.name == 'nt' and len(sys.argv) <= 1:
+        print("usage: drag-and-drop a path script .txt onto `Update_Paths.bat`\n  or")
+
+    parser = argparse.ArgumentParser(prog=prog, description=desc, formatter_class=argparse.RawTextHelpFormatter)
     parser.add_argument("update_source", help="update file path / line of update notation")
     parser.add_argument("--map", "-m", help="path to update, defaults to all maps in maps directory")
     parser.add_argument("--directory", "-d", help="path to maps directory, defaults to _maps/")

tools/UpdatePaths/readme.md

@@ -0,0 +1,32 @@
+HOW TO USE:
+Drag one of the scripts in the “Scripts” folder onto the .bat file “Update_Paths” to open it with the .bat file. Let the script run.
+
+Use this tool before using mapmerge or opening the map in an editor.
+
+IMPORTANT:
+Please tie the script you are making to the associated PR on github and put it in the scripts folder when you are done.
+
+For example: 17980_rust_converter ties into https://github.com/ParadiseSS13/Paradise/pull/17980
+
+HOW TO MAKE A SCRIPT:
+This tool updates paths in the game to new paths. For instance:
+If you have a path labeled
+
+/obj/structure/door/airlock/science/closed/rd
+
+and wanted it to be
+
+/obj/structure/door/airlock/science/rd/closed
+
+This tool would update it for you! This is extremely helpful if you want to be nice to people who have to resolve merge conflicts from the PRs that you make updating these areas.
+
+How do you do it?
+Simply open a notepad and type this on a line:
+
+/obj/structure/door/airlock/science/closed/rd : /obj/structure/door/airlock/science/rd/closed
+
+The path on the left is the old, the path on the right is the new. It is seperated by a ":"
+If you want to make multiple path changes in one script, simply add more changes on new lines.
+
+If you get lost, look at other scripts for examples.
+

tools/bootstrap/python

@@ -0,0 +1,118 @@
+#!/bin/sh
+# bootstrap/python
+#
+# Python-finding script for all `sh` environments, including Linux, MSYS2,
+# Git for Windows, and GitHub Desktop. Invokable from CLI or automation.
+#
+# If a python.exe installed by `python_.ps1` is present, it will be used.
+# Otherwise, this script requires a system `python3` and `pip` to be provided,
+# and will create a standard virtualenv in which to install `requirements.txt`.
+set -e
+
+# Convenience variables
+Bootstrap="$(dirname "$0")"
+Sdk="$(dirname "$Bootstrap")"
+Cache="$Bootstrap/.cache"
+if [ "$TG_BOOTSTRAP_CACHE" ]; then
+	Cache="$TG_BOOTSTRAP_CACHE"
+fi
+OldPWD="$PWD"
+cd "$Bootstrap/../.."
+. ./_build_dependencies.sh  # sets PYTHON_VERSION
+cd "$OldPWD"
+PythonVersion="$PYTHON_VERSION"
+PythonDir="$Cache/python-$PythonVersion"
+PythonExe="$PythonDir/python.exe"
+Log="$Cache/last-command.log"
+
+# If a portable Python for Windows is not present, search on $PATH.
+if [ "$(uname)" = "Linux" ] || [ ! -f "$PythonExe" ]; then
+	# Strip the "App Execution Aliases" from $PATH. Even if the user installed
+	# Python using the Windows Store on purpose, these aliases always generate
+	# "Permission denied" errors when sh.exe tries to invoke them.
+	PATH=$(echo "$PATH" | tr ":" "\n" | grep -v "AppData/Local/Microsoft/WindowsApps" | tr "\n" ":")
+
+	# Try to find a Python executable.
+	if command -v python3 >/dev/null 2>&1; then
+		PythonExe=python3
+	elif command -v python >/dev/null 2>&1; then
+		PythonExe=python
+	elif command -v py >/dev/null 2>&1; then
+		PythonExe="py -3"
+	else
+		echo
+		if command -v apt-get >/dev/null 2>&1; then
+			echo "Please install Python using your system's package manager:"
+			echo "    sudo apt-get install python3 python3-pip"
+		elif [ "$(uname -o)" = "Msys" ]; then
+			echo "Please run tools/bootstrap/python.bat instead of tools/bootstrap/python once to"
+			echo "install Python automatically, or install it from https://www.python.org/downloads/"
+			# TODO: give MSYS pacman advice?
+		elif command -v pacman >/dev/null 2>&1; then
+			echo "Please install Python using your system's package manager:"
+			echo "    sudo pacman -S python python-pip"
+		else
+			echo "Please install Python from https://www.python.org/downloads/ or using your system's package manager."
+		fi
+		echo
+		exit 1
+	fi
+
+	# Create a venv and activate it
+	PythonDir="$Cache/venv"
+	if [ ! -d "$PythonDir" ]; then
+		echo "Creating virtualenv..."
+		"$PythonExe" -m venv "$PythonDir"
+	fi
+	if [ -f "$PythonDir/bin/python" ]; then
+		PythonExe="$PythonDir/bin/python"
+	elif [ -f "$PythonDir/scripts/python3.exe" ]; then
+		PythonExe="$PythonDir/scripts/python3.exe";
+	else
+		echo "bootstrap/python failed to find the python executable inside its virtualenv"
+		exit 1
+	fi
+fi
+
+# Use pip to install our requirements
+if [ ! -f "$PythonDir/requirements.txt" ] || [ "$(b2sum < "$Sdk/requirements.txt")" != "$(b2sum < "$PythonDir/requirements.txt")" ]; then
+	echo "Updating dependencies..."
+	"$PythonExe" -m pip install -U wheel
+	"$PythonExe" -m pip install -U pip -r "$Sdk/requirements.txt"
+	cp "$Sdk/requirements.txt" "$PythonDir/requirements.txt"
+	echo "---"
+fi
+
+# Verify version and deduce the path separator
+PythonMajor=${PythonVersion%%.*}
+PythonMinor=${PythonVersion#*.}
+PythonMinor=${PythonMinor%.*}
+PATHSEP=$("$PythonExe" - "$PythonMajor" "$PythonMinor" <<'EOF'
+import sys, os
+if sys.version_info.major != int(sys.argv[1]) or sys.version_info.minor < int(sys.argv[2]):
+	print("Error: Python ", sys.argv[1], ".", sys.argv[2], " or later is required, but you have:\n", sys.version, sep="", file=sys.stderr)
+	exit(1)
+print(os.pathsep)
+EOF
+)
+
+# Cheap shell function if tee.exe is not available
+if ! command -v tee >/dev/null 2>&1; then
+	tee() {
+		# Fudge: assume $1 is always "-a"
+		while read -r line; do
+			echo "$line" >> "$2"
+			echo "$line"
+		done
+	}
+fi
+
+# Invoke python with all command-line arguments
+export PYTHONPATH="$Sdk$PATHSEP${PYTHONPATH:-}"
+mkdir -p "$Cache"
+printf '%s\n' "$PythonExe" "$@" > "$Log"
+printf -- '---\n' >> "$Log"
+exec 4>&1
+exitstatus=$({ { set +e; "$PythonExe" -u "$@" 2>&1 3>&-; printf %s $? >&3; } 4>&- | tee -a "$Log" 1>&4; } 3>&1)
+exec 4>&-
+exit "$exitstatus"

tools/bootstrap/python.bat

@@ -0,0 +1,2 @@
+@echo off
+call powershell.exe -NoLogo -ExecutionPolicy Bypass -File "%~dp0\python_.ps1" %*

tools/bootstrap/python37._pth

@@ -0,0 +1,5 @@
+python37.zip
+.
+..\..\..
+
+import site