Skip to content

Latest commit

 

History

History
195 lines (141 loc) · 8.86 KB

1.1-QuickStarted.md

File metadata and controls

195 lines (141 loc) · 8.86 KB

Getting Started

Development Approach

MaaFramework supports complete low-code programming through JSON (Pipeline JSON) and also provides APIs for developers to integrate on their own.
It is also possible to combine both approaches, using low-code as a form of "wrapper" for invocation.
Below are several common integration methods:

Complete Dependence on JSON Low-Code Programming (Using MaaPiCli.exe)

This method is simple and quick, but not very flexible. It is recommended for beginners and those new to programming with MaaFramework.
We provide a 🎞️ video tutorial and ⭐ project template for this method. Here is an example:

// JSON does not support comments; this is pseudo-code for reference only and cannot be executed directly
{
    "Recognize and Click Start Button": {
        "recognition": "OCR",           // Optical Character Recognition
        "expected": "Start",           // Text to recognize
        "action": "Click",             // Action: Click
        "next": [
            "Recognize and Click Confirm Icon"
        ]
    },
    "Recognize and Click Confirm Icon": {
        "recognition": "TemplateMatch", // Image template matching
        "template": "Confirm.png",       // Image file name
        "action": "Click"
    }
}

Use JSON Low-Code Programming with Custom Logic for Complex Tasks

Launch the CLI through the API while registering some custom tasks. This method allows a seamless switch from method 1. Here is an example:

{
    "Recognize and Click Confirm Icon": {
        "next": [
            "My Custom Node"
        ]
    },
    "My Custom Node": {
        "recognition": "Custom",
        "custom_recognition": "MyReco",
        "action": "Custom",
        "custom_action": "MyAct"
    }
}
# This is pseudo-code for reference only and cannot be executed directly
def main():
    # Register custom recognition
    maafw.Toolkit.register_custom_recognition("MyReco", MyRecognition())

    # Register custom action
    maafw.Toolkit.register_custom_action("MyAct", MyAction())

    # Start MaaPiCli
    maafw.Toolkit.run_pi_cli("C:/MaaXXX/resource", "C:/MaaXXX/cache")

class MyRecognition(CustomRecognition):
    def analyze(context, ...):
        # Obtain the image and perform custom image processing
        image = context.tasker.controller.cached_image
        # Return image analysis result
        return AnalyzeResult(box=(10, 10, 100, 100))

class MyAction(CustomAction):
    def run(context, ...):
        # Perform click action
        context.controller.post_click(100, 10).wait()
        # Override the next tasks to execute
        context.override_next(node_name, ["TaskA", "TaskB"])

Write Your Own Code

You can use low-code as a "wrapper" for invocation or register custom callbacks.

# This is pseudo code, for reference only, and cannot be run directly
# "Recognize and click the start button", "Recognize and click the confirmation icon" and so on are all logic in Json
def main():
    detail = tasker.post_task("Recognize and click the start button").wait().get()

    if detail.completed:
        tasker.controller.post_click(100, 100).wait()

    else:
        image = tasker.controller.cached_image
        save_to_file(image)

        tasker.resource.register_custom_action("MyAction", MyAction())
        tasker.post_task("Recognize and click the confirmation icon").wait()

    image: np.ndarray = tasker.controller.post_screencap().wait().get()

Prepare resource files

⭐If you use the project template, just modify it directly in folder.

You need to prepare some resource files with the typical file structure as follows:

my_resource
├── image
│   ├── my_image_1.png
│   └── my_image_2.png
├── model
│   └── ocr
│       ├── det.onnx
│       ├── keys.txt
│       └── rec.onnx
└── pipeline
    ├── my_pipeline_1.json
    └── my_pipeline_2.json

You can modify the names of files and folders starting with "my_", but the others have fixed file names and should not be changed. Here's a breakdown:

Pipeline JSON Files

The files in my_resource/pipeline contain the main script execution logic and recursively read all JSON format files in the directory.

You can refer to the Task Pipeline Protocol for writing these files. You can find a simple demo for reference.

Tools:

  • JSON Schema
  • VSCode Extension
    • Config resources based on interface.json
    • Support going to task definition, finding task references, renaming task, completing task, click to launch task
    • Support launching as MaaPiCli
    • Support screencap and crop image after connected

Image Files

The files in my_resource/image are primarily used for template matching images, feature detection images, and other images required by the pipeline. They are read based on the template and other fields specified in the pipeline.

Please note that the images used need to be cropped from the lossless original image and scaled to 720p. If you use an Android emulator, please use the screenshot function that comes with the emulator! (You cannot directly take screenshots of the emulator window)

UNLESS YOU EXACTLY KNOW HOW MAAFRAMEWORK PROCESSES, DO USE THE CROPPING TOOLS BELOW TO OBTAIN IMAGES.

Text Recognition Model Files

⭐If you use the project template, just follow its documentation and run configure.py to automatically deploy the model file.

The files in my_resource/model/ocr are ONNX models obtained from PaddleOCR after conversion.

You can use our pre-converted files: MaaCommonAssets. Choose the language you need and store them according to the directory structure mentioned above in Prepare Resource Files.

If needed, you can also fine-tune the official pre-trained models of PaddleOCR yourself (please refer to the official PaddleOCR documentation) and convert them to ONNX files for use. You can find conversion commands here.

Debug

  • Use MaaDebugger.

  • Use VSCode plugin.

  • If you use MaaPiCli, the config/maa_option.json file will be generated in the same directory, including:

    • logging: Save the log and generate debug/maa.log. Default true.
    • recording: Save recording function, which will save all screenshots and operation data during operation. You can use DbgController for reproducible debugging. Default false.
    • save_draw: Saves the image recognition visualization results. All image recognition visualization results drawings during the run will be saved. Default false.
    • show_hit_draw: Displays the node hit pop-up window. Each time the recognition is successful, a pop-up window will display the recognition result. Default false.
    • stdout_level: The console displays the log level. The default is 2 (Error), which can be set to 0 to turn off all console logs, or to 7 to turn on all console logs.

  • If you integrate it yourself, you can enable debugging options through the Toolkit.init_option / MaaToolkitConfigInitOption interface. The generated json file is the same as above.

Run

You can integrate MaaFramework using Generic CLI (MaaPiCli), third-party Generic GUI (MFAWPF, etc) or by writing integration code yourself.

Using MaaPiCli

⭐If you use the project template, follow its documentation directly and run install.py to automatically package the relevant files.

Use MaaPiCli in the bin folder of the Release package, and write interface.json and place it in the same directory to use it.

The Cli has completed basic function development, and more functions are being continuously improved! Detailed documentation needs to be further improved. Currently, you can refer to Sample to write it.

Examples:

Writing Integration Code Yourself

Please refer to the Integration Documentation.