Classes - accessibilitysoftwarehub/OpenSourceWindowsGazeControl GitHub Wiki

Decription of each class and how it is used

CustomFixationDataStream

Purpose: Provide stream of data that contains information about where a user's gaze is fixating on screen.

How it works: This class is driven on a typical coordinate data stream that comes from an eye tracker containing a x and y coordinate and a timestamp. As Events are fired from the stream of normal coordinates a ring bugger of the x and y coordinates is continually filled. At each event The average gaze location is worked out as well as the variance from the average. When the average is subtracted from the variance it gives the value of the standard deviation, If the standard deviation is 0 for the x axis the current gaze is not moving horizontally the same is true for the y axis except it means the gaze is not moving vertically. The standard deviation of the gaze is checked against a threshold, if below that threshold on both axis, the users eyes are typically not moving it it can be determined that user is starting to fixate on an area on the screen. If the beginning of a fixation is detected, the CustomFixationDataStream will move into a Beginning Fixation state and raise and event with the average gaze location coordinates, a timestamp, and the type of CustomFixationEvent which is EFixationStreamEventType.Start. If the standard deviation of the gaze stays below the threshold the the CustomFoxationDataStream will continue to raise events with the fixation type of EFixationStreamEventType.Middle, This continues until the gaze moves and the standard deviation goes above the set threshold. From these events fixations can be determined, this work is carried out by the FixationDetection class.

The CustomFixationClass can be tuned by changing the values of xFixationThreshold, yFixationThreshold and Buffersize. Lowering the fixation the Threshold values makes it less likely for fixations to be detected. Lowering the buffersize makes fixations quicker to detect but also as this data is used for drawing current onscreen gaze, it makes this animation jerky.

FixationDetection

Purpose: Transform information for CustomFixationDatastream into usable fixation Events.

How it works: FixationDetection subscribes the 'next' event from CustomFixationDataStream. From the stream of events FixationDetection calculates if there is a fixation based on the type of fixation event that is raised by CustomFixationDataStream and how long since the type of event was a EFixationStreamEventType.Start event without being interrupted by a EFixationStreamEventType.End event. If the time limit is reached without being interrupted by EFixationStreamEventType.End a flag is set in the state manager that a fixation has occurred, the StateManager then knows to get the coordinates and proceed with its next piece of work.

Fixation detection is initiated by the StateManager calling the StartDetectingFixation() method of FixationDetection. This changes the fixationState of FixationDetection to EFixationState.DetectingFixation, and starts TimeOutTimer which is used to stop fixation detection if the user takes too long to fixate on something, it also acts a safety if a fixation function is accidentally activated and the user is unable to cancel out of it. Once FixationDetection is in EFixationState.DetectingFixation it starts checking the types of events being raised by CustomfixationDataStream, if a EFixationStreamEventType.Start event occurs a timer called fixationTimer is started, each next event can be either a middle or end event. If it its a middle event onFixationProgressEvent() is raised which is an event that other objects can subscribe to track the progress of a fixation (currently used by GazeHighlight for user feedback about where they are gazing and how fixation progress is going), also the property's xPosFixation and yPosFixation are updated. This continues until one of the following conditions are meet.

There is a CustomFixationDetectionStream EFixationStreamEventType.End, when this happens Fixation detection stops the fixation timer and goes back to waiting for a EFixationStreamEventType.Start event.
The TimeOutTimer reaches its Elapsed value because no fixation has been reached by the value stored in FixationTimeOutLength property, when this happens TimeOutTimer raises its elapsed event which fixationTimeOut() is subscribed to.

The fixationTimer reaches the time set in FixationDetectionTimeLength and raises its elapsed event, the method runActionWhenTimerReachesLimit() is subscribed to this, it stops the timeOutTimer, sets FixationDetections fixationState back to EFixationState.WaitingForFixationRequest and raises SystemFlags.Gaze with the StateManager, the StateManager now knows there has been a successful fixation and calls getXY() which returns the coordinates of the last successful fixation.

KeyboardHook

Purpose: Capture Keyboard button presses even when the applications main form does not have focus.

How it works: KeyboardHook utilities methods from user32.dll to hook into the windows api. It does this by creating and inserting a delegate that points back to method in the managed CLI code of the KeyboardHook Class, in this case LowlevelKBhookDelegate points at method HookCallBack. When eve HookCallBack is called it uses a Marshaller to convert unmanaged data in lParam back to something the local CLI can use which is an int that represents a keycode. When this happens the OnkeyPresesed event is raised and the vkCode is converted into a Key passed via a HookedKeyboardEvent. Any method subscribed to OnkeyPressed event receives the last key the was pressed on the keyboard. It is important when utilizing this class to remove the hook via UnHookKeyboard() method when the host application closes, as it could potentially leave windows in an unstable condition as the hook is in a series chain and if not removed may stop other hooks from receiving key presses.

ShortcutKeyWorker

Purpose: Monitor keyboard key presses for keys assigned to GazeTooBar functions, if a assigned key is pressed signal the statemanager to run the associated function.

How it works When instantiated ShortCutkeyWorker subscribes to the KeyBoardHooks OnkeyPressed event, it is also passed in a dictionary that maps functions of the GazeToolBar to keyboard keys. When OnKeypressed event happens the RunKeyFunction() method is called, this converts the key pressed into a string, which is passed into a series of if statements (couldn't use a switch here as c# required a constant value) to check if the pressed key is assigned to a function. The mapped keys and functions are stored in a Dictionary. The Enum ActioToBePerformed is used as key in a dictionary which is used to associate a function with a key, as these Enums are always going to be the same and cover all the functions available, this also allows the associated keys to be changed. This way no matter what key is assigned to a function of the tool bar it can always be linked back to a GazeToolBar function. If the key is found to mapped to a function two thing happen, flags are raised to interrupt the state machine and transitions into the requested function and the coordinates of the users gaze are recorded. The state machine uses these coordinates to show a zoom window and then proceed with the requested function.

PointSmoother

NOTE: currently not used, as the CustomFixationDataStream has the same smoothing effect on the data, can be implement for extra smoothing by changing bool usePointSmoother to true in the FixationDetection Class.

Purpose: To collect Gaze coordinates into a ring buffer and from these points return the average gaze location which smooths the data and removes the jumping around that appears in a raw gaze data stream..

How it works: When PointSmoother is instantiated it is given a integer which represent how big the buffer is, as UpdateAndGetSmoothPoint() is called, the buffer is filled up, and a smoother point relative to the previous call is passed back. The larger the buffer the smoother the points are but also the slow gaze responds to to changes.

ScrollControl

Purpose: Track a users gaze and computer this into actions that scroll the in-focus window up, down, left and right.

How it Works: When instantiated Scroll Control is passed in a value for duration between scroll steps and a scalar value, these are used to calculate the overall speed and acceleration of scrolling movement. The deadzone percentages are used to calculate an area in the middle of the screen so that when gaze falls in that region no scrolling is done. The method SetDeadZoneBounds() calculates and creates a deadzone relative to the the size of the screen and passed in deadzone percentages, this dead zone is stored in a NoScrollRect Struct. ScrollControl subscribes the method updateGazeCoodinates() to a gaze data stream, it updates local fields that store the xy location, it also runs a method that checks if a users gaze is off screen and raises a flag with the StateManager to transition out of scroll mode if they are outside the screen bounds. The actual scrolling is run by a timer called scrollStepTimer, on its elapse event the scroll() method is run, which checks if the either the currentGazeLocationX or currentGazeLocationY coordinate are outside the deadZoneRect, if they are the coordinate is passed into calculateScrollSpeed() which calculates the direction and how many steps to scroll based on how far outside the DeadZoneRect the users gaze is. Once these values have been calculated they are passed into VirtualMouse.Scroll() which translates these values in to middle mouse wheel scroll clicks, and if a windows is in focus it will scroll in the direction of the users gaze.

VirtualMouse

Purpose: Simulate functions a physical mouse provides, left, right, double, and middle click. Scroll and move mouse position.

How it works: VirtualMouse utilises methods from user32.dll and windowsInput.dll. Each pubic method takes a xy cooridate, its uses this coordinate to first call SetCursorPos which moves the on screen mouse cursor, then the actual function of the method is call eg left click. Note that windowsinput.dll is was compiled to include a MiddleMouseButton click, this function is not avaliable in other versions found on the internet more info can be found here External libraries

GazeKeyboard

Purpose: Creates the keyboard and returns it

How it works: OptikeyLite uses a WPF forms Window object to create the keyboard. The services that allow OptikeyLite to accept input, simulate output and make predictive text suggestions are created and provided to the keyboard. The WindowManipulationServcie is then created which gives options for resizing, re positioning, changing opacity and state. This uses persistent settings found in settings under OptikeyLite properties. The window is then returned.

ZoomLens

Purpose: Zooms in on an area of the screen for precised clicking

How it works: The ZoomLens is a windows form which is used to create a magnified portion of the screen and translates the user's gaze into accurate on-screen coordinates. The magnified area is based on where the user has gazed, the user can then gaze on the zoomed-in form and the coordinates are translated back to the original desktop coordinates. This class can also reposition itself to account for edges and corners of the screen. It detects when the user is looking near an edge and will reposition the zoomed in area and window to prevent the window going off screen and to allow the user to click on the very edge of the screen.

StateManager

Purpose: Manages System States

How it works: The state machine is similar to a Finite State Machine (FSM) This class controls the flow of the program by keeping track of various states and flags. It works by running the update() and action() method in an infinite loop. The update() method controls what state the system should be in and determines if the system should change states. After the update() method the action() method is called, this method executes the necessary actions at each state e.g during the zooming state, the StateManager instantiates the ZoomLens class in order to execute the zooming function.