Winter 2021

Screenland - A Short History

- "the under appreciated touch input"

Recently I left my “smart” watch on the table to charge overnight beside my “smart” phone. They have some sort of relationship since they are of the same species and to my surprise the next morning the watch had acquired several new functions such as the local weather report, various “notifications” and access to my e-mail as well. To get these watch displays, I had to "swipe" the screen with my fingertip. Since the watch is the “little sister” of the phone, I was curious to see how well its screen performed in comparison to that of the phone. It is a strange thing to interact with a watch via a touch screen. It is also a strange thing to do to a phone. Why is everybody looking down at a thin glass rectangle but also stroking it or seemingly massaging it with their thumbs? So, for those of you who have recently woken from a long nap, this is the story. Like most technology in our lives, we did not have any of this 25 years ago.

Much has been written recently about cell phone addiction and how it seems to take over so much of some peoples’ lives. We would need some help from the social sciences to really understand this behaviour - perhaps to be covered in a later article. For now, I will try to explain a little more about the device itself and maybe why it is so appealing. First the image. There is no doubt that it must “move.” Why were “movies” virtually an instant hit in the 1900s? Then W.D. Griffiths discovered that showing a face close-up captivated audiences (for example, Lilian Gish 1919). About the same time, Edison produced the Nickelodeon that used a film strip projected in a box. Thus was born movies for personal viewing (1905). It was another 100 years before movies could be viewed again for personal viewing on our phones (today streamed over a high-speed data communications channel). Early film projection flickered, which is why they were nicknamed the “Flicks.” We don’t want this, so quite early a film frame rate of 24 frames/sec was standardized. In the TV era we learned to be conscious of image resolution in terms of TV scan lines. The North American system used 512 horizontal lines across the screen, so we never sat too close to the TV console.

The first flat screen monitor was invented in 1964, a single-colour plasma type, then the first full-colour plasma screen TV in 1977. Later, LED technology was introduced in 1997. Screens were characterized by picture size and pixel horizontal x vertical count. So how much density do we really need? It turns out from the print industry that we need 300 dots/inch or better. At this density the individual dots cannot be perceived by one with normal vision. Even when aided by a 10X magnifier, one could barely resolve the dots. I suggest using a 20X magnifier or a jeweller’s loop if you would like to check your smartphone and iPad display.

Apple is well aware of this and in its advertising has branded this quality “retina” a display. For an example of a current smart phone, the iPhone X - 2436 x 1125 pixels for a density of 458 dots/inch. The pixel centre-to-centre distance equals 0.055mm. For a point of comparison, if you are interested in spacing of the photoreceptors (cones) in the fovea (central part) of our retinas, it is approximately 0.003mm and there are 150,000 cones/sq. mm. The Apple iPhone “retina” is 20x less dense than the human retina!

The construction of the screen on our smartphones appears simple but is a modern miracle of electronic technology and is composed of two basic parts laminated together.

Touch screen evolution

Development starts with the capacitive touch-sensitive switch. You will find these today on the call panels for elevators, on consumer appliances and on medical equipment.

If you remember what capacitance is, just skip ahead. A capacitor can be formed by two close-together parallel conducting metal plates separated by a dielectric layer, or just air. You can transfer an electronic charge to it, which results in a voltage appearing across the plates. Other electrical circuitry can measure or do something with this voltage. If you touched a second capacitor to it, some charge would redistribute to it changing the voltage slightly. The surface of a human body can act as one plate of a capacitor and the ground another. The capacitance value is not large, and we used to have all sorts of fun charging ourselves up. We don’t have to purposely charge ourselves up, we are all walking-around little capacitors. It is quite easy to charge up a capacitor by touch if its capacitance value is much smaller than our body capacitance. If you want to design a touch switch, you use a plate to touch, some amplifier circuitry and a relay to mechanically open or close a single circuit. In the ‘70s you could package this in something about the size of a paperclip box and to provide some feedback, remember to incorporate a small indicator light.

The air traffic control problem

In the ‘60s and ‘70s, commercial air traffic controllers were seeing up to 100 small radar blips on their screens. They did not have modern cursors or track balls to direct the computer’s attention to a specific target. The first capacitive touch screen to solve this problem was an overlay semi-transparent grid of crossed wires in the x and y axis so that each intersection point formed a small capacitor. The grid was aligned with the radar screen. All the operators had to do was to “tap” the point of interest. This is the first application of the development and use of a capacitive touch panel. Gradually touch panels found their way into all kinds of kiosks, e.g. ATM machines and many types of appliances. It is easy to find many installations of touch panels in our environment today.

To be complete, there are two other technologies that can be used to form panels. For example, there is a good chance that the controls on your electrical range are touch enabled but require some mechanical force to activate. This is the so-called “resistive” type. The reason is that you don’t want to accidentally brush over a very sensitive capacitive-technology switch and unintentionally turn on the range. However, today more than 90 % of touch screens use capacitive technology.

Smartphones

So, getting to the touch screens for phones. In 2007, Apple bought a company called FingerWorks for its touch technology and applied for a patent that described how the technology would be used to “scroll” through a display list on its iPod player.

In another Apple patent filed in 2008 we learned about the implementation of multi-touch capability and the recognition of a few simple gestures. These were “Swipe” and “Pinch,” pioneered in the first iPads and iPhones. We are very experienced in using these finger commands and they are almost universally incorporated in all brands of devices today.

There are three common ways to hold and operate a smartphone today. Hold it in a single hand and use that thumb to scroll; hold it in one hand and use the index finger of the second hand to tap on a screen area you want; hold it two hands and use both thumbs to tap the virtual keyboard. The phone’s operating system recognizes these modes of use and optimizes how it responds.

The touch screen sensing array

If you take a closer look, you will see that one axis of the crossed x-y grid is formed by very thin parallel semitransparent metal strips etched in just on the under surface of the covering glass. Next is an insulting layer and then a second crossed strip layer, then various semiconductor layers. A critical characteristic is that about 90% of the light from below gets through.

In the early days for tablets, people wanted a touch point per display pixel. Has this been achieved?

Well, not quite anymore. Apple specifies 264 ppi, which is like a spacing of 0.1mm. Compare this to 458 dots/in for its display.

For a human reference, you may be curious about how tightly packed the pressure sensitive neuron receptors (Merkel receptors) in our fingertips are. So, each efferent nerve is responsible for a receptor field just below the skin of about five sq mm subdivided into eight irregular regions. The Merkel receptors in these regions are spaced as tight as 0.5 mm.

Processing the sensory information is quite different. In our case the brain has outsourced some processing of the sensing neural networks so that they can pre-process for edge detection and roughness.

For our smartphone, the capacitive sensing array outputs a charge “map.” It is the software’s job to recognize blobs as finger taps, one or several at the same time, then find the centroids and track and report its motion and direction across the screen. Fortunately, because we now have powerful processors operating in nano seconds, the phone’s response can make this operation appear to us smooth and fluid. Except occasionally when it doesn’t, and we might curse at our phones.

So, the next time you are mindlessly scrolling through photos or e-mails, just give a moment’s thought to what is happening just below the surface in the smartphone and our fingertips.

                                                            Ron Miller, illustrations by Susan Schwartz