Added new metrics module

Added a metrics module to SR, which allows the user to implement their own metrics. This module will give the user all the information about the simulation (using the criteria and the CARLA recorder), so that they can implement any metrics they desire. The recorder at scenario_runner.py has been modified. It now requires a path (relative to SCENARIO_RUNNER_ROOT), where the CARLA recorder will be saved. It also saves the criteria, dumping all the JSON serializable attributes into a .json file.
carla-simulator · Sep 7, 2020 · 7432727 · 7432727
1 parent 47bf022
commit 7432727
Show file tree

Hide file tree

Showing 24 changed files with 3,712 additions and 29 deletions.
diff --git a/Docs/CHANGELOG.md b/Docs/CHANGELOG.md
@@ -25,7 +25,8 @@
 * The new weather parameters (related to fog) are now correctly read when running scenarios outside routes.
 * Enable weather animation during scenario execution (requires ephem pip package)
 * Changed manual control to be in par with the CARLA version. Among others, added vehicle lights, recording and some new sensors
-* Removed unsupported scenarios (ChallengeBasic and BackgroundActivity, VehicleTurnLeftAtJunction) 
+* Removed unsupported scenarios (ChallengeBasic and BackgroundActivity, VehicleTurnLeftAtJunction)
+* Added a new metrics module, which gives access to all the information about a scenario in order to allow the user to extract any desired information about the simulation. More information [here](metrics_module.md)
 * OpenSCENARIO support:
     - Added support for controllers and provided default implementations for vehicles and pedestrians. This required changing the handling of actors, which results in that now all actors are controlled by an OSC controller. Supported controllers:
         - Pedestrian controller

diff --git a/Docs/img/metrics_example.jpg b/Docs/img/metrics_example.jpg
diff --git a/Docs/img/metrics_example.jpg~ b/Docs/img/metrics_example.jpg~
diff --git a/Docs/img/metrics_example.png b/Docs/img/metrics_example.png
diff --git a/Docs/index.md b/Docs/index.md
@@ -25,15 +25,17 @@ CARLA forum</a>
         — Brief tutorials on how to run different types of scenarios.  
     [__Create a new scenario__](creating_new_scenario.md)
         — Tutorial on how to create a new scenario using ScenarioRunner.  
-    [__F.A.Q.__](faq.md)
+    [__Metrics module__](metrics_module.md)
+        — Explanation of the metrics module.  
+    [__F.A.Q.__](FAQ.md)
         — Some of the most frequent installation issues.  
-    [__Release notes__](release_notes.md)
+    [__Release notes__](CHANGELOG.md)
         — Features, fixes and other changes listed per release.  
   </p>
 
 ## References
   <p style="padding-left:30px;line-height:1.8">
-    [__List of scenarios_](list_of_scenarios.md)
+    [__List of scenarios__](list_of_scenarios.md)
         — Example scenarios available in ScenarioRunner.  
     [__OpenScenario support__](getting_started.md)
         — Support status of OpenSCENARIO features.  

diff --git a/Docs/metrics_module.md b/Docs/metrics_module.md
diff --git a/Docs/ros_agent.md b/Docs/ros_agent.md
@@ -25,7 +25,7 @@ Derive from RosAgent and implement the sensors() method.
         def sensors(self):
             return [ <sensor-definition> ]
 
-As an example for the sensor definition, see [HumanAgent.py](../srunner/autoagents/HumanAgent.py).
+As an example for the sensor definition, see [HumanAgent.py](../srunner/autoagents/human_agent.py).
 
 
 ### Define Startup

diff --git a/metrics_manager.py b/metrics_manager.py
@@ -0,0 +1,154 @@
+#!/usr/bin/env python
+
+# Copyright (c) 2020 Computer Vision Center (CVC) at the Universitat Autonoma de
+# Barcelona (UAB).
+#
+# This work is licensed under the terms of the MIT license.
+# For a copy, see <https://opensource.org/licenses/MIT>.
+
+# Allows the execution of user-implemented metrics
+
+"""
+Welcome to the ScenarioRunner's metric module
+
+This is the main script to be executed when running a metric.
+It is responsible of parsing all the information and executing
+the metric specified by the user.
+"""
+
+import os
+import sys
+import importlib
+import inspect
+import json
+import argparse
+from argparse import RawTextHelpFormatter
+
+import carla
+from srunner.metrics.tools.metrics_log import MetricsLog
+
+
+class MetricsManager(object):
+    """
+    Main class of the metrics module. Handles the parsing and execution of
+    the metrics.
+    """
+
+    def __init__(self, args):
+        """
+        Initialization of the metrics manager. This creates the client, needed to parse
+        the information from the recorder, extract the metrics class, and runs it
+        """
+        self._args = args
+
+        # Parse the arguments
+        recorder_str = self._get_recorder(self._args.log)
+        criteria_dict = self._get_criteria(self._args.criteria)
+
+        # Get the correct world and load it
+        map_name = self._get_recorder_map(recorder_str)
+        world = self._client.load_world(map_name)
+        town_map = world.get_map()
+
+        # Instanciate the MetricsLog, used to querry the needed information
+        log = MetricsLog(recorder_str)
+
+        # Read and run the metric class
+        metric_class = self._get_metric_class(self._args.metric)
+        metric_class(town_map, log, criteria_dict)
+
+    def _get_recorder(self, log):
+        """
+        Parses the log argument into readable information
+        """
+
+        # Get the log information.
+        self._client = carla.Client(self._args.host, self._args.port)
+        recorder_file = "{}/{}".format(os.getenv('SCENARIO_RUNNER_ROOT', "./"), log)
+
+        # Check that the file is correct
+        if recorder_file[-4:] != '.log':
+            print("ERROR: The log argument has to point to a .log file")
+            sys.exit(-1)
+        if not os.path.exists(recorder_file):
+            print("ERROR: The specified log file does not exist")
+            sys.exit(-1)
+
+        recorder_str = self._client.show_recorder_file_info(recorder_file, True)
+
+        return recorder_str
+
+    def _get_criteria(self, criteria_file):
+        """
+        Parses the criteria argument into a dictionary
+        """
+        if criteria_file:
+            with open(criteria_file) as fd:
+                criteria_dict = json.load(fd)
+        else:
+            criteria_dict = None
+
+        return criteria_dict
+
+    def _get_metric_class(self, metric_file):
+        """
+        Function to extract the metrics class from the path given by the metrics
+        argument. Returns the first class found that is a child of BasicMetric
+
+        Args:
+            metric_file (str): path to the metric's file.
+        """
+        # Get their module
+        module_name = os.path.basename(metric_file).split('.')[0]
+        sys.path.insert(0, os.path.dirname(metric_file))
+        metric_module = importlib.import_module(module_name)
+
+        # And their members of type class
+        for member in inspect.getmembers(metric_module, inspect.isclass):
+            # Get the first one with parent BasicMetrics
+            member_parent = member[1].__bases__[0]
+            if 'BasicMetric' in str(member_parent):
+                return member[1]
+
+        print("No child class of BasicMetric was found ... Exiting")
+        sys.exit(-1)
+
+    def _get_recorder_map(self, recorder_str):
+        """
+        Returns the name of the map the simulation took place in
+        """
+
+        header = recorder_str.split("\n")
+        sim_map = header[1][5:]
+
+        return sim_map
+
+
+def main():
+    """
+    main function
+    """
+
+    # pylint: disable=line-too-long
+    description = ("Scenario Runner's metrics module. Evaluate the execution of a specific scenario by developing your own metric.\n")
+
+    parser = argparse.ArgumentParser(description=description,
+                                    formatter_class=RawTextHelpFormatter)
+    parser.add_argument('--host', default='127.0.0.1',
+                        help='IP of the host server (default: localhost)')
+    parser.add_argument('--port', '-p', default=2000,
+                        help='TCP port to listen to (default: 2000)')
+    parser.add_argument('--log', required=True,
+                        help='Path to the CARLA recorder .log file (relative to SCENARIO_RUNNER_ROOT).\nThis file is created by the record functionality at ScenarioRunner')
+    parser.add_argument('--metric', required=True,
+                        help='Path to the .py file defining the used metric.\nSome examples at srunner/metrics')
+    parser.add_argument('--criteria', default="",
+                        help='Path to the .json file with the criteria information.\nThis file is created by the record functionality at ScenarioRunner')
+    # pylint: enable=line-too-long
+
+    args = parser.parse_args()
+
+    MetricsManager(args)
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -11,6 +11,7 @@ nav:
   - Get ScenarioRunner: getting_scenariorunner.md
   - First steps: getting_started.md
   - Create a new scenario: creating_new_scenario.md
+  - Metrics module: metrics_module.md
   - FAQ: FAQ.md
   - Release Notes: CHANGELOG.md
 - References:

diff --git a/requirements.txt b/requirements.txt
@@ -7,4 +7,5 @@ carla
 ephem
 tabulate
 opencv-python==4.2.0.32
-numpy
+numpy
+matplotlib
diff --git a/scenario_runner.py b/scenario_runner.py
@@ -27,6 +27,7 @@
 import signal
 import sys
 import time
+import json
 import pkg_resources
 
 import carla
@@ -40,7 +41,7 @@
 from srunner.tools.route_parser import RouteParser
 
 # Version of scenario_runner
-VERSION = 0.6
+VERSION = '0.9.9'
 
 
 class ScenarioRunner(object):
@@ -175,7 +176,6 @@ def _cleanup(self):
             settings.fixed_delta_seconds = None
             self.world.apply_settings(settings)
 
-        self.client.stop_recorder()
         self.manager.cleanup()
 
         CarlaDataProvider.cleanup()
@@ -255,6 +255,38 @@ def _analyze_scenario(self, config):
             if not (self._args.output or filename or junit_filename):
                 print("Please run with --output for further information")
 
+    def _record_criteria(self, criteria, name):
+        """
+        Filter the JSON serializable attributes of the criterias and
+        dumps them into a file. This will be used by the metrics manager,
+        in case the user wants specific information about the criterias.
+        """
+        file_name = name[:-4] + ".json"
+
+        # Filter the attributes that aren't JSON serializable
+        with open('temp.json', 'w') as fp:
+
+            criteria_dict = {}
+            for criterion in criteria:
+
+                criterion_dict = criterion.__dict__
+                criteria_dict[criterion.name] = {}
+
+                for key in criterion_dict:
+                    if key != "name":
+                        try:
+                            key_dict = {key: criterion_dict[key]}
+                            json.dump(key_dict, fp, sort_keys=False, indent=4)
+                            criteria_dict[criterion.name].update(key_dict)
+                        except TypeError:
+                            pass
+
+        os.remove('temp.json')
+
+        # Save the criteria dictionary into a .json file
+        with open(file_name, 'w') as fp:
+            json.dump(criteria_dict, fp, sort_keys=False, indent=4)
+
     def _load_and_wait_for_world(self, town, ego_vehicles=None):
         """
         Load a new CARLA world and provide data to CarlaDataProvider
@@ -352,17 +384,23 @@ def _load_and_run_scenario(self, config):
             return False
 
         try:
-            # Load scenario and run it
             if self._args.record:
-                self.client.start_recorder("{}/{}.log".format(os.getenv('SCENARIO_RUNNER_ROOT', "./"), config.name))
+                recorder_name = "{}/{}/{}.log".format(
+                    os.getenv('SCENARIO_RUNNER_ROOT', "./"), self._args.record, config.name)
+                self.client.start_recorder(recorder_name, True)
+
+            # Load scenario and run it
             self.manager.load_scenario(scenario, self.agent_instance)
             self.manager.run_scenario()
 
             # Provide outputs if required
             self._analyze_scenario(config)
 
-            # Remove all actors
+            # Remove all actors, stop the recorder and save all criterias (if needed)
             scenario.remove_all_actors()
+            if self._args.record:
+                self.client.stop_recorder()
+                self._record_criteria(self.manager.scenario.get_criteria(), recorder_name)
 
             result = True
 
@@ -457,45 +495,50 @@ def main():
     main function
     """
     description = ("CARLA Scenario Runner: Setup, Run and Evaluate scenarios using CARLA\n"
-                   "Current version: " + str(VERSION))
+                   "Current version: " + VERSION)
 
+    # pylint: disable=line-too-long
     parser = argparse.ArgumentParser(description=description,
                                      formatter_class=RawTextHelpFormatter)
+    parser.add_argument('-v', '--version', action='version', version='%(prog)s ' + VERSION)
     parser.add_argument('--host', default='127.0.0.1',
                         help='IP of the host server (default: localhost)')
     parser.add_argument('--port', default='2000',
                         help='TCP port to listen to (default: 2000)')
+    parser.add_argument('--timeout', default="10.0",
+                        help='Set the CARLA client timeout value in seconds')
     parser.add_argument('--trafficManagerPort', default='8000',
                         help='Port to use for the TrafficManager (default: 8000)')
     parser.add_argument('--sync', action='store_true',
                         help='Forces the simulation to run synchronously')
-    parser.add_argument('--debug', action="store_true", help='Run with debug output')
+    parser.add_argument('--list', action="store_true", help='List all supported scenarios and exit')
+
+    parser.add_argument(
+        '--scenario', help='Name of the scenario to be executed. Use the preposition \'group:\' to run all scenarios of one class, e.g. ControlLoss or FollowLeadingVehicle')
+    parser.add_argument('--openscenario', help='Provide an OpenSCENARIO definition')
+    parser.add_argument(
+        '--route', help='Run a route as a scenario (input: (route_file,scenario_file,[route id]))', nargs='+', type=str)
+
+    parser.add_argument(
+        '--agent', help="Agent used to execute the scenario. Currently only compatible with route-based scenarios.")
+    parser.add_argument('--agentConfig', type=str, help="Path to Agent's configuration file", default="")
+
     parser.add_argument('--output', action="store_true", help='Provide results on stdout')
     parser.add_argument('--file', action="store_true", help='Write results into a txt file')
     parser.add_argument('--junit', action="store_true", help='Write results into a junit file')
     parser.add_argument('--outputDir', default='', help='Directory for output files (default: this directory)')
-    parser.add_argument('--waitForEgo', action="store_true", help='Connect the scenario to an existing ego vehicle')
+
     parser.add_argument('--configFile', default='', help='Provide an additional scenario configuration file (*.xml)')
     parser.add_argument('--additionalScenario', default='', help='Provide additional scenario implementations (*.py)')
+
+    parser.add_argument('--debug', action="store_true", help='Run with debug output')
     parser.add_argument('--reloadWorld', action="store_true",
                         help='Reload the CARLA world before starting a scenario (default=True)')
-    # pylint: disable=line-too-long
-    parser.add_argument(
-        '--scenario', help='Name of the scenario to be executed. Use the preposition \'group:\' to run all scenarios of one class, e.g. ControlLoss or FollowLeadingVehicle')
+    parser.add_argument('--record', type=str, default='',
+                        help='Path were the files will be saved, relative to SCENARIO_RUNNER_ROOT.\nActivates the CARLA recording feature and saves to file all the criteria information.')
     parser.add_argument('--randomize', action="store_true", help='Scenario parameters are randomized')
     parser.add_argument('--repetitions', default=1, type=int, help='Number of scenario executions')
-    parser.add_argument('--list', action="store_true", help='List all supported scenarios and exit')
-    parser.add_argument(
-        '--agent', help="Agent used to execute the scenario (optional). Currently only compatible with route-based scenarios.")
-    parser.add_argument('--agentConfig', type=str, help="Path to Agent's configuration file", default="")
-    parser.add_argument('--openscenario', help='Provide an OpenSCENARIO definition')
-    parser.add_argument(
-        '--route', help='Run a route as a scenario (input: (route_file,scenario_file,[number of route]))', nargs='+', type=str)
-    parser.add_argument('--record', action="store_true",
-                        help='Use CARLA recording feature to create a recording of the scenario')
-    parser.add_argument('--timeout', default="10.0",
-                        help='Set the CARLA client timeout value in seconds')
-    parser.add_argument('-v', '--version', action='version', version='%(prog)s ' + str(VERSION))
+    parser.add_argument('--waitForEgo', action="store_true", help='Connect the scenario to an existing ego vehicle')
 
     arguments = parser.parse_args()
     # pylint: enable=line-too-long

diff --git a/srunner/metrics/data/CriteriaFilter.log b/srunner/metrics/data/CriteriaFilter.log