Screen Time Feature - gnh1201/welsonjs GitHub Wiki

Screen Time Feature

Starting from WelsonJS version 0.2.7.34, a feature called 'Screen Time' is supported.

This was introduced to enable integration with highly restrictive applications. It is useful in cases where integration methods like APIs are not supported, and the status can only be checked visually on the computer screen.

This feature does not allow operation in the background. It only functions in User Interactive mode, where users can monitor the progress.

How to Use

1. Configure the settings.ini file

In the settings.ini file, configure the service settings by adding the following:

[Service]
DISABLE_SCREEN_TIME=true
; backward,save,sample_width=128,sample_height=128,sample_adjust_x=0,sample_adjust_y=0,sample_any=button.png:message.png:like.png,sample_ocr=like.png,sample_clipboard=button.png,sample_nodup=message.png,process_name=notepad.exe
SCREEN_TIME_PARAMS=save
  • DISABLE_SCREEN_TIME=true: Disables the screen time feature.
  • SCREEN_TIME_PARAMS=save: Enables the screenshot saving function. Screenshots will be saved to the directory app/assets/img/_captured.

The commented-out line provides an example of more advanced parameters you can use, such as image searches, OCR, clipboard copying, and duplication prevention.

2. Place templates in the app/assets/img/_templates directory

  • Add the image fragments (templates) you want to match in PNG format in this directory.
  • If you want to match the images in grayscale (ignoring color), prefix the file names with binary_. This helps in scenarios where color information isn't necessary for matching.

3. Run the service in User Interactive mode

  • To start the service, run the startInteractiveService.bat file. This will enable interactive processing of templates on the screen.

4. Handle template match events

  • Note: As of WelsonJS version 0.2.7.37, the onServiceScreenTime() event has been renamed to onScreenTemplateMatched().
  • When a template (image fragment) is matched, the onScreenTemplateMatched() event will be triggered.
  • This event provides the matched template's name and the coordinates where it was found. You can use this information to perform actions based on the match.
  • Example usage can be found in defaultService.example.js.

5. Optimize with the onScreenNextTemplate() event

  • You can use the onScreenNextTemplate() event to improve processing speed by specifying the next template image to be handled.
  • If you return the name of the next image to be processed, only that image will be considered, and the rest will be ignored, leading to faster execution.
  • If you do not specify this event, all images will continue to be processed.

Available options

  • backward: This option means that the search for the matching location using the template image will be performed in reverse order. The search will proceed from the bottom-right corner to the top-left corner of the current computer screen.

  • save: This option saves the entire screenshot to a specific directory (app/assets/img/_captured).

  • sample_width={sample_width} and sample_height={sample_height}: These define the size of the sample region for the image search, setting it to a {sample_width}x{sample_height} pixel area.

  • sample_adjust_x=0 and sample_adjust_y=0: These options adjust the position of the sample area horizontally (x) and vertically (y). A value of 0 means no adjustment; the sample area is positioned exactly at the detected coordinates.

  • sample_any=button.png:message.png:like.png: This specifies a list of image templates (button.png, message.png, like.png) that the process will try to match in the specified region. It will stop as soon as one of these is found.

  • sample_ocr=like.png: Once like.png is found in the search, OCR (optical character recognition) will be performed around the detected area, specifically within a {sample_width}x{sample_height} pixel region. The OCR is powered by Tesseract OCR.

  • sample_clipboard=button.png: When button.png is detected, a {sample_width}x{sample_height} pixel region around the detection is copied to the clipboard.

  • sample_nodup=message.png:

    • When message.png is found for the first time, it is added to a queue for tracking.

    • If the same image fragment is detected again, the system checks the queue to see if it has already been processed.

    • If the image is in the queue, it skips processing it again, effectively preventing duplicate actions. However, the system continuously checks for new detections to ensure that only the first occurrence is processed.

  • process_name=notepad.exe: This specifies the process or application (in this case, Notepad) in which the image search and operations will be performed.