Guitar As An Introduction to Bimodal Control Of Software
Consider how the guitar exists as an instrument.
The hands are assigned separate tasks.
The left hand navigates to the note and sets it up to be played.
It moves across the fretboard before pressing down on the string(s)
The right hand triggers the note's sound
It holds the guitar pick and picks the string(s) according to the timing.
The two activities are interdependent. For the note to be played
it has to be set up by the left hand. The left hand relies on
the right hand to play the note or it will remain silent.
The Mouse
Noctivagous explains that the mouse's control of desktop computer software
should seek out the same principles. As the mouse is used today,
it places too much cognitive load on the computer user by
combining 1) steering
(moving the cursor) and 2) triggering of functionality (clicking things)
into a single device and location. It assigns all of that to one hand.
Making these roles distinct, separating them into two input locations that use both hands,
provides the user greater control over software. The trackpad can, for example,
continue to steer the cursor while the "clicks" occur elsewhere, controlled by
the other hand for the first time.
An altogether different software dynamic emerges, as shown in the demo below.
For the software developer the opportunities in UI theory open up.
Importantly, just one person devised the mouse in 1968, Douglas Engelbart,
and it has remained unchanged since that time.
Its introduction was
an upgrade to the personal computer at the time, but decades later,
in many instances, it produces a tedious and tiresome way to work with
a computer. Touchscreen tablets did not make it irrelevant. For one,
straight-on touch of a screen object is not precise.
There is much room for improvement for both style of interaction with
a computer and they can benefit from bimodal control / dual-input theory.
Vector-Drawing Canvas Demo
Experience this UI theory by making shapes with the vector-drawing demo below
then dragging them around. In keeping with bimodal control theory, this
uses the keyboard keys to click instead of the mouse buttons.
After moving your cursor into the gray drawing area below, use the following
keys.
There is also an interactive keyboard inside the drawing area that shows what keys can be pressed.
DRAWING:
1. Press the F key to deposit a point.
2. Move the mouse.
3. Press the F key
again to deposit another point.
4. Repeat until you wish to press an END key:
A - END the stroke path.
E - END and fill the shape.
R - END and stroke the shape.
S - END and fill+stroke the shape.
SELECTING AND DRAGGING:
1.Select objects with Tab.
2.To drag the selection, turn on the drag-lock with SPACEBAR,
then move the mouse. (Press it again to turn it off.)
3. Stamp a selection with W while dragging.
4. Scale a selection down and up with [ and ].
5. Rotate a selection forwards and backwards by 15 degrees with ; and '.
6. The Esc key will also cancel the selection and dragging.
ADJUSTING STROKE WIDTH:
1. To thin or thicken the stroke, use C and V.
Description of Bimodal Control UI Theory
In the vector-drawing demo above, the mouse and keyboard
were given separate roles.
The mouse for cursor steering. The mouse was given the role of steering the cursor and
navigation.
The keyboard for function activation. The keyboard was given the role of activation ("click").
Here, each input device operates in its own "modality," focusing on a specific type of interaction.
Previously the two were conjoined into one device, for just one hand.
Division of Tasks
Bimodal Control UI theory separates into two physical locations:
steering, navigating, and guiding activity
e.g. cursor XY(Z) movement (via mouse or 3D controller). the movement of the crosshair, etc.
triggering of functions
e.g. mouse button "clicks", tapping an object on a touchscreen, pressing a button on an XR controllers
Importantly, these are still fused today and have not been separated. But the benefits are evident in the vector-drawing demo.
More Bimodal Control UI Examples
Importantly, Bimodal Control UI Theory is not limited to
the specific context of changing the desktop computer UI by moving mouse clicks to the keyboard keys.
That is the interactivity of a desktop computer hardware setup, altered in
one way that can be explained by bimodal control theory.
Bimodal Control UI theory is actually the intentional separation
of input tasks; the user shifts between actions in separate
device locations. The effect is the alternation of input activity in two
locations. We provide some other examples below.
Dual-Input Web Hyperlink Browsing
In some instances, the dual-input setup
that represents bimodal control theory may
not feature a mouse or XY steering device
at all.
In NPSurfer,
a web surfing plug-in, avoids using
the mouse altogether for activating web page links.
Instead, a 3x3 grid is overlayed on the page when
the user decides navigation to links should occur. The division of
the page with this 3x3 grid corresponds to the keys in the number pad.
For this plug-in, there is a separation keyboard tasks for to both hands.
One hand selects which region the link is located by
way of a key that corresponds to the grid (3x3 grid
divides the web page temporarily and corrrespond to number keys).
The other hand activates the links labels that
show up within that specific cell of the grid.
Dual-Input Tablet Hardware Input
For an alternative touchscreen tablet UI, Noctivagous
has discussed plans for a touchscreen paired with a column
of LED-backed buttons to the left of the screen.
The dual-input occurs by holding down one or more of the
LED-screen-backed buttons that sits to the left
of the touchscreen, then pressing the screen.
One hand either momentarily presses or holds down the specific LED-screen-backed key(s) to the left
of the touchscreen.
The other hand presses the object or objects on the touchscreen
which are the recipients/target(s) of that button's function.
Deleting an object then involves pressing the "Delete" button
to the left of the touchscreen on the tablet with one hand
while touching the objects with the right hand.
Dual-Input Manipulation of Text Fields and Sliders
In Noctivagous software, the mouse is used to hover
over the textfield. The textfield becomes outlined
and its textfield is highlighted, and then the user begin typing
on top (to replace) what is in the textfield immediately
without having to clicking inside the textfield to
put a cursor inside.
A second functionality built in is that if the
user hovers over a numeric textfield with the cursor
and presses "[" or "]", the textfield value will decrement or
increment. Againt, the user does not click inside
the textfield. It only requires hovering over
it with the cursor and then "[" or "]" are pressed,
(since these characters are not in any value of the
numeric textfield). Pressing Shift+"[" or Shift+"]"
will decrement or increment by a higher value, and pressing
option will be a smaller value.
This also works for slider controls: hovering over the slider
and pressing "[" or "]" will decrement or increment the
value.
The Bimodal Control UI Theory Can Improve All Software UI
Important to mention about the Bimodal Control UI Theory
is that once it is established in software, it carries implications
s across the whole of desktop software. It can be applied
to many specific conventions of desktop software, such as
node-based programming in which the programmer connects nodes.
The nodes can be connected with lines rapidly or detached
from the nodes with greater control by the user because they
use two inputs categorized into steering and activation.
The nodes themselves can be dragged around in the same
way as the drag-lock in the vector-drawing demo.
More About The Vector-Drawing Demo
The demo above is limited to drawing straight lines, but a programmer
can use the same vector line segments as the basis for:
the diameter of a circle.
the radius of a circle.
the diagonal of a rectangle.
the radius of a circle inscribing a regular polygon.
points of an arc
points of a spline
etc.
In Floating Marker,
curves have been implemented with these straight segments through the
b-spline (similar to Bezier paths). In combination with the dual-input UI,
Noctivagous uses its own data structure
made of composite spline points that bow (bend) the line, as shown in the video below.
More Than Just for Making Vector Graphics
For the software developer, there are other
uses of the dual-input UI are possible besides making vector-graphics.
The vector line can serve as:
a polyline (polygonal shape) that selects objects inside (whether they are files or something else).
the means of connecting nodes in node-based/patch-based programming.
this has already been tested and is far superior to using the mouse for making node scripting diagrams
the diagonal of the live selection rectangle (such as when selecting files on a desktop).
This is provided feature for selecting objects in the vector-drawing program Floating Marker.
In the same way that vector shapes are dragged, the drag-lock can be used to move windows
on the screen. A different OS GUI can come about: the scale buttons allow the user to shrink windows.
It is possible to implement an entire desktop UI in the dual-input UI,
with no mouse clicks.
Relationship to History
During the original 1968 demo
of the mouse by Douglas Engelbart, there was
a five-key device sitting to the left of the keyboard.
The mouse was placed to the right of the keyboard.
This five-key device was later dropped at Xerox PARC for the Alto, which set
the model for the Apple Lisa and Macintosh,
but it was included as part of Engelbart's setup.
In professional media production software today,
such as video editors and 3D modeling programs,
the mouse is used to simulate a hand
on the screen, but that would be a single finger, only able to
click one button at a time.
The dual-input UI breaks away from this confining setup and repurposes
the keyboard keys as a set of buttons for a machine. Using it feels
more like interacting with the computer mechanically because
the keyboard and mouse are reconfigured to provide mechanical controls.
This is more appropriate for software, especially applications
like video editors where there are constant interactions
with mechanisms, like the timeline. Mechanisms pervade user interface
but have always been regarded as as something more vague in experience.
For example, the scrollbar is a mechanism. The system menu
is also a machine containing menu items that drop down.
For the future of software, this dual-input is the best modification for
software running on desktop computers as they are owned by consumers across the world,
if the goal is the match a mechanical controls with the user interface.
The dual-input UI can provide the core UI component of
a new type of software, and this demo on this page
is just what is provided as an introduction, so it isn't
going to address every aspect all at once.
There are other points to discuss about implementing
it comprehensively, such as that text fields, sliders,
and everything else should be modified to match what
it provides.
Once practiced for a period of time, the dual-input UI becomes
natural and preferred over the standard use of the mouse
for software.
The dual-input UI Comes from Floating Marker, a Vector-Drawing App
An app under development, called Floating Marker, is the origin of
the dual-input UI. For the benefit of developers who use GitHub,
the dual-input UI is demonstrated on this page. Its applicability
extends beyond vector-drawing to software situations of all kinds.
It can take over the positioning of windows, the dragging of nodes in node-based programming,
the connecting of nodes (with lines), and the changing of values in sliders and other controls.
All desktop computers are equipped with
keyboard and mouse, and the dual-input UI is a convenient upgrade for interactive
vector graphics software.
Full Description of the dual-input UI
The dual-input UI is an improved software user interface for the desktop computer.
A software program that implements the dual-input UI treats the keys
on a keyboard as click buttons. The mouse buttons are ignored except for
the purpose of accommodating the old habits that all users have.
After a person becomes accustomed to the dual-input UI, he or she will
want it implemented across all of desktop user interfaces.
The concept is that there is currently too much work assigned to an individual mouse hand.
Compared to the dual-input UI, the mouse hand is currently overstressed.
It has to do two things at once: coordinate the position of the cursor and click the buttons with the index
finger.
In a vector-drawing program, especially, this is too much; the
standard mouse interface is too simplistic for drawing vectors. Thus, to
establish a normal situation for
drawing vectors with a mouse, it's important first to modify the overall interface.
Separating the click from the mouse divides the labor of
directing cursor activity to two hands instead of one. This leads to a leap in possible dexterity.
Each hand has been given its own role: the mouse hand just moves the cursor and
the clicking hand just clicks with keys. This opens up a totally different interactive
dynamic with the desktop computer, because previously there was never such alternation occuring between
two hands (move, click, move, click). But importantly, many more "click" buttons are available,
because any alphanumeric key can serve as one.
Implementing Selection And Dragging of Objects in The dual-input UI
The Tab Key Is Used for The Selection Click
As shown in the demo above, to select an individual object, you press the
tab key, and it is selected immediately underneath the cursor
as if you had pressed shift+left mouse button. The
tab key
selects and de-selects. It is the selection
click.
Though any key could be used for this, tab
has been chosen because of its location on the keyboard
that makes the keyboard behave as live controls instead
of character entry for a program.
The purpose of assigning shift+left mouse button
to the tab key is that the dual-input UI works better with the shift modifier for selections
most of the time. The interactive dynamic has changed, and it is far easier to select
multiple objects at a time with the dual-input UI.
Tweaks and modifications like this are needed to make the dual-input UI translate existing conventions.
Just like before, clicking outside of the selection de-selects everything.
The Spacebar Implements a Drag-Lock on The Current Selection
In a convention specific to the dual-input UI, the tab key has been paired with the spacebar
key to move objects around the screen. It is possible to use any combination of keys, but
these are very large and they will be tapped frequently as buttons of a machine.
To drag an object, press
the spacebar key after the object has been selected (using tab click), and the
object moves in lock step with the cursor. This is a drag lock, affixing objects to the
cursor relative to their position at the time of the drag lock. To release the drag lock, press the
spacebar key again
and then the objects are left at their location.
For ease of providing upfront terminology this is called a drag-lock, but it is also called
"carting" by Noctivagous, in place of "dragging". It is named as such because a cart moves its contents
smoothly using wheels on the ground whereas the dragging of something on the ground makes lots of noise.
That is the degree of UI improvement that carting (a drag-lock) provides.
The Escape Key Cancels All Drag Locks And Selections
If the Escape is pressed and there is any drag lock occurring, it will
be turned off. The current selection will be canceled as well.
Using the Escape for this is not only a convenience, it fits into the whole
of using of the keyboard to click when using the dual-input UI.
The Opportunities That Arise When The Drag Lock Is Used
Moving objects around on screen is very smooth when a drag lock is
used in the way described above. But in addition to this, it provides
opportunities to operate with the selection or apply functions to it
in real time.
In a vector-drawing program, the drag-locked selection can be scaled up or down with
the bracket keys ([ and ]).
In the demo, the two keys below the bracket keys (; and ')
rotate the selection by an increment of 15 degrees clockwise and counterclockwise.
The selection can be stamped
with the E key during the drag-lock. So this means that transformed copies of
the dragged selection can occur in real time. This is what Floating Marker uses for
its vector-drawing user interface.
