Interface ScreenAPI

interface ScreenAPI {
    GetPixelColor(x: number, y: number): string;
    FindImage(
        imagePath: string,
        threshold?: number,
        region?: WindowBounds,
    ): null | ImageMatchResult;
    DebugCapture(): string;
    FindImageBase64(
        base64Data: string,
        threshold?: number,
        region?: WindowBounds,
    ): null | ImageMatchResult;
}

Methods

GetPixelColor FindImage DebugCapture FindImageBase64

Methods

GetPixelColor

  • GetPixelColor(x: number, y: number): string

    Returns the color of a single pixel on screen as a hex string.

    Parameters

    • x: number

      Pixel X coordinate.

    • y: number

      Pixel Y coordinate.

    Returns string

    Hex color string, e.g. "#FF0000" for red.

    Example

    const color = Screen.GetPixelColor(100, 200);
    if (color === "#FF0000") Console.Log("red pixel");

    Requires

    screen

FindImage

  • FindImage(
        imagePath: string,
        threshold?: number,
        region?: WindowBounds,
    ): null | ImageMatchResult

    Searches the screen for a template image loaded from the script asset sandbox or an explicitly user-selected file path. Uses masked normalized cross-correlation: transparent pixels in the template PNG are ignored, so only opaque regions (text, borders) contribute to the match score.

    Parameters

    • imagePath: string

      Script asset path, or an absolute path selected by a UI file/directory picker.

    • Optionalthreshold: number

      Minimum correlation score to consider a match, between 0 and 1. Defaults to 0.8.

    • Optionalregion: WindowBounds

      Search region { x, y, width, height } in virtual screen coordinates. Omit to search all monitors.

    Returns null | ImageMatchResult

    An ImageMatchResult with the center coordinates and score, or null if no match meets the threshold.

    Remarks

    ALWAYS check for null before accessing .x or .y. The return value is null when no match meets the threshold. imagePath values are resolved canonically. Paths inside the script asset sandbox are accepted, and .. escapes are denied. Absolute paths outside the sandbox are accepted only when the path was selected by the user through UI.AddFilePicker or is inside a directory selected through UI.AddDirectoryPicker. Use FindImageBase64 when embedding a template directly in a script. USE CASE: Pass a region from Window.Find() bounds to restrict the search area; this is significantly faster on multi-monitor setups.

    Example

    const template = UI.AddFilePicker("save_button", "Save button template", { filters: ["png"] });
    const match = Screen.FindImage(template.GetValue());
    if (match) {
    Mouse.Move(match.x, match.y);
    Mouse.Click();
    }

    // Restrict search to a specific window's area
    const [win] = Window.Find({ process: "chrome.exe" });
    const match2 = Screen.FindImage(template.GetValue(), 0.9, win.bounds);

    Requires

    screen

DebugCapture

  • DebugCapture(): string

    Captures the current screen and saves it to the debug directory. Useful for diagnosing why FindImage cannot find a template.

    Returns string

    The file path where the screenshot was saved.

    Example

    const path = Screen.DebugCapture();
    Console.Log("Saved to: " + path);

    Requires

    screen

FindImageBase64

  • FindImageBase64(
        base64Data: string,
        threshold?: number,
        region?: WindowBounds,
    ): null | ImageMatchResult

    Searches the screen for a template image provided as Base64-encoded PNG data. Transparent pixels are masked out; only opaque regions contribute to the score.

    Parameters

    • base64Data: string

      Base64-encoded PNG image data with transparent background.

    • Optionalthreshold: number

      Minimum correlation score to consider a match, between 0 and 1. Defaults to 0.8.

    • Optionalregion: WindowBounds

      Search region { x, y, width, height } in virtual screen coordinates. Omit to search all monitors.

    Returns null | ImageMatchResult

    An ImageMatchResult with the center coordinates and score, or null if no match meets the threshold.

    Remarks

    ALWAYS check for null before accessing .x or .y. USE CASE: Embed template images directly in the script without relying on external files. Encode the PNG to Base64 once and store as a constant.

    Example

    const B64_ICON = "iVBORw0KGgo..."; // your Base64 PNG
    const match = Screen.FindImageBase64(B64_ICON, 0.85);
    if (match) {
    Mouse.Move(match.x, match.y);
    Mouse.Click();
    }

    Requires

    screen

Settings

Member Visibility

On This Page

Methods