One run should log the following info for both BASE and MROS

[darkobozhinoski]

Log this info:

• distance to the goal (already logged);
• time the mission is above safety violation - N/A;
• time the mission is above energy violation - N/A;
• percentage of the mission above safety violation;
• percentage of the mission above energy violation;
• time to complete;
• boolean if the goal is reached (1);
• boolean if the quality of the mission is satisfied (2)
  (report true only and only if (i) safety is above the safety violation less than 5% of the time; (ii) energy is above the violation less than 10% of the time). Otherwise, report false;
• boolean if the average safety is satisfied (3). First, remove all values for safety equal to 0. Then, calculate the average safety on the rest of the data. (report true if average safety level < 0.5 (This is an example number))
• CPU Usage;
• Memory consumption;

The booleans (1,2,3) will help us define if a mission run was successful. If any of these values report false, we can say that the mission failed. Otherwise, it succeeds. Definition of what is a successful run for the robot here: https://docs.google.com/document/d/1pGxXg7ierV7L0mMmBsZvgysEYl14Xm1b_GXzoPDJK7g/edit#bookmark=id.3pygfcy1yfbp

[marioney]

@darkobozhinoski A couple questions.

[ ] time to complete;

Do you mean run time? I mean the total time of the mission?

• time the mission is above safety violation;

Do you need the actual time, or is it enough to have the percentage?

[darkobozhinoski]

@darkobozhinoski A couple questions.

[ ] time to complete;

Do you mean run time? I mean the total time of the mission?

Yes, that Is correct - the total time of the mission.

• time the mission is above safety violation;

Do you need the actual time, or is it enough to have the percentage?

I guess to calculate the percentage you need to know how long the robot was under a certain violation in one run. As I remember, we already collected that info in the first submission (it should be there).

[marioney]

I guess to calculate the percentage you need to know how long the robot was under a certain violation in one run. As I remember, we already collected that info in the first submission (it should be there).

Actually, it's easier for me to count only # of times, that way it's independent of the log frequency. I can get the percentage from there

[darkobozhinoski]

Ok, we can remove the average violation time in seconds. We should have enough data to show, without that.

[marioney]

in 3845a92 I've added the CPU usage and memory consumption

[darkobozhinoski]

Additional list of information that needs to be logged @marioney

• Average for safety/energy risk level;
• Number of times the energy margin was violated;
• Number of times the safety was violated;
• Mission successful

We make a distinction between safety margin and critical safety threshold. You need to define the critical safety threshold.

[marioney]

* How many times the critical threshold is crossed;

@darkobozhinoski what is this critical value? didn't we say it should never be crossed?

[marioney]

Number of times the threshold was violated;
Number of times the safety was violated;

@darkobozhinoski Aren't those two the same?

[darkobozhinoski]

@marioney about the critical value, we say that it should never be crossed. The idea is coming from Carlos if I remember well and the point is that with BASE it might be the case that the critical value is crossed, while the Metacontrol does not allow that.
The number of times refers to the number of times the threshold was violated (we refer to all thresholds). Safety should be also checked independetly.