As of RuneLite Launcher 1.5.2, RuneLite uses its own DPI settings. Before making any changes, make sure you have installed the latest RuneLite Launcher. If you are unsure, simply reinstall the launcher. If that did not work, try the following.

Overriding DPI scaling

(If "System (enhanced)" didn't have the desired effect, try also "System" and "Application")

High DPI sharp/integer scaling

To avoid the client looking "blurry" it's recommended to enable High DPI Integer scaling:

• in the windows "RuneLite Properties" High DPI settings {"High DPI scaling override": "Application"}
• enable stretched mode integer scaling
• enable stretched mode increased performance mode
• set gpu plugin UI Scaling Nearest Neighbor