Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REG-2180] Add monkey bot action space analysis results #353

Merged
merged 8 commits into from
Nov 14, 2024

Conversation

nAmKcAz
Copy link
Collaborator

@nAmKcAz nAmKcAz commented Nov 8, 2024

  • Adds end of bot analysis results for action manager bots use of actions
    • This used actions data is actually captured on a PER TICK basis if we wanted to add a usedActions report for each tick at some future point
    • Captures the invocation count and objects targeted for each used action
"usedActions": [
        {
            "metrics": {
                "apiVersion": 26,
                "invocations": 80,
                "objectsActedOn": [
                    "CharacterSelectCanvas/PlayerSeats/PlayerSeat (4)/AnimationContainer/ClickInteract//Button",
                    "CharacterSelectCanvas/PlayerSeats/PlayerSeat (5)/AnimationContainer/ClickInteract//Button",
                    "CharacterSelectCanvas/PlayerSeats/PlayerSeat (3)/AnimationContainer/ClickInteract//Button"
                ]
            },
            "action": {
                "actionTypeName": "RegressionGames.ActionManager.Actions.UIButtonPressAction, RegressionGames, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null",
                "paths": [
                    "Unity UI/Button/BossRoom.Client.UICharSelectPlayerStateBox.OnClicked"
                ],
                "objectTypeName": "UnityEngine.UI.Button, UnityEngine.UI, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null",
                "parameterRange": {
                    "type": "RANGE_BOOL",
                    "minValue": 0,
                    "maxValue": 1
                },
                "eventListenerName": "BossRoom.Client.UICharSelectPlayerStateBox.OnClicked"
            }
        },
        ...
]
  • Captures the action information for all unused actions
"unusedActions":[
        {
            "actionTypeName": "RegressionGames.ActionManager.Actions.MouseButtonAction, RegressionGames, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null",
            "paths": [
                "Unity.BossRoom.Gameplay.Actions.AoeActionInput/AoeActionInput.cs/Mouse.current.rightButton"
            ],
            "objectTypeName": "Unity.BossRoom.Gameplay.Actions.AoeActionInput, Unity.BossRoom.Gameplay, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null",
            "parameterRange": {
                "type": "RANGE_BOOL",
                "minValue": 0,
                "maxValue": 1
            },
            "mouseButtonFunc": {
                "funcType": "TYPE_CONSTANT",
                "data": "1"
            }
        },
        ...
]
  • Fixes a bug around async file writes on metadata during end recording
  • slight performance improvements to metadata file conversions at end of recording

Find the pull request instructions here

Every reviewer and the owner of the PR should consider these points in their request (feel free to copy this checklist so you can fill it out yourself in the overall PR comment)

  • The code is extensible and backward compatible
  • New public interfaces are extensible and open to backward compatibility in the future
  • If preparing to remove a field in the future (i.e. this PR removes an argument), the argument stays but is no longer functional, and attaches a deprecation warning. A linear task is also created to track this deletion task.
  • Non-critical or potentially modifiable arguments are optional
  • Breaking changes and the approach to handling them have been verified with the team (in the Linear task, design doc, or PR itself)
  • The code is easy to read
  • Unit tests are added for expected and edge cases
  • Integration tests are added for expected and edge cases
  • Functions and classes are documented
  • Migrations for both up and down operations are completed
  • A documentation PR is created and being reviewed for anything in this PR that requires knowledge to use
  • Implications on other dependent code (i.e. sample games and sample bots) is considered, mentioned, and properly handled
  • Style changes and other non-blocking changes are marked as non-blocking from reviewers

@nAmKcAz nAmKcAz marked this pull request as ready for review November 13, 2024 17:00
@nAmKcAz nAmKcAz requested a review from vontell as a code owner November 13, 2024 17:00
@nAmKcAz nAmKcAz requested review from batu and abeizer November 13, 2024 17:00
@nAmKcAz
Copy link
Collaborator Author

nAmKcAz commented Nov 13, 2024

This is step 1.5 in getting to where we have the data to drive more 'intelligent' exploration bots for games.

Copy link
Contributor

@batu batu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good overall to me, but I would love to see a video overall as I haven't really seen it in action. (I believe Sasha finished these right around the time I joined)

It also occurred to me now that Runespawn can be used in conjunction botSequences, I can pass in int segmentNumber in Simulation interface.

Copy link
Collaborator

@vontell vontell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very excited to see this get in! The ability to automatically gather more useful data during a monkey test will be very valuable, especially as we start to add more and more to this foundation.

I'm especially excited to see what ideas folks have on the code coverage part - that information could be really valuable (i.e. pointing the developer to parts of the code in a nice UI or format related to a particular validation).

During our 1:1 it'd be great to dive deeper into any observations and ideas you have. Curious to hear what potential next steps would be.

@@ -270,6 +280,7 @@ private async Task HandleEndRecording(long tickCount,
_needToRefreshAssets = true;
#endif

//TODO (REG-2181): Add upload logic for metadata like action analysis, code coverage, etc - we probably also want consider if we redo the game metadata upload here and roll these into a single zip file of metadata ?
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I wonder if it might get messy with the number of zip files we are uploading now... something to think about.

@nAmKcAz nAmKcAz merged commit 120818a into main Nov 14, 2024
2 checks passed
@nAmKcAz nAmKcAz deleted the zack/post-run-analysis-data branch November 14, 2024 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants