Enabling Floating TIP for Java Swing Apps on Tablet PCs (Part 1)

I became a Tablet PC user this past summer when I acquired a TabletKiosk i7210 eo (an Ultra Mobile PC) running Windows XP Tablet Edition. My primary job is as a Java developer and I have done a fair amount of development using Java’s Swing windowing toolkit.

I very quickly learned that apps written using this windowing toolkit don’t work like regular windows apps on a tablet PC. They don’t offer access to the floating text entry window (Floating TIP) forcing the user to enter text via the text input panel at the bottom of the screen, a more cumbersome method of entry.

The Floating TIP

When running a normal Windows app on a tablet PC and you put focus in a text field/area/panel that doesn’t offer ink-based text entry support (most apps don’t), a small button will appear hovering near the cursor. This button is used to open the floating Tablet Input Panel (TIP) which lets Tablet PC users enter text in a variety of ways without the use of a physical keyboard.

For instance, here is the floating TIP button on the start run text entry field:

When clicked, the floating TIP appears near where the button used to be and the user can enter text via the on-screen keyboard or via a couple methods of handwriting.



Java Swing Text Entry – A 3rd rate experience

Anytime you have to enter text on a tablet PC using something other than “inking,” many tablet PC users consider it a second rate experience having to use the floating TIP. Well when a text field isn’t recognized by Windows at all, you have to use the non-floating TIP at the bottom of the screen, what I’d consider to be a third rate experience.

Here are the issues:

Issue #1 – Accessing the TIP Button is Not Always Convenient

First you click in the field you want to enter positioning the caret where you want to insert text or selecting text you want to delete or over type. Next you find the TIP button on the Windows task bar next to the Start button.

Since the UMPC is using a rather smallish touch screen, I’m limited to 1024×600 resolution. So I have configured the task bar to not stay on top of other windows so that I can get that last little bit of screen space. It is very common for one or more windows to be covering up the task bar. So I’ve got to minimize or move windows to see the task bar. Alternatively, I mapped one of the hardware buttons to the Start menu so I can pop it open just to bring the entire start bar to the front.

Issue #2 – Non-floating TIP wastes space

When you click the TIP button, it opens an onscreen menu that’s takes up 15-25% of your screen and forces all applications that are visible to shrink to fit in the area above it. This wastes space while it is visible. And if you’re using an app where all text entry needs to be done via the non-floating TIP, you basically end up keeping it open most of the time.



Worse however, is when you close the TIP panel, Windows doesn’t always resize your windows to the sizes they were before the TIP was opened. It seems it will do it properly for the window that had focus when you opened the TIP if that same window has focus when you close the TIP. But all other windows remained shortened. And it never resizes Java applications to their pre-TIP sizes, at least for me.

But Why is Java Swing Broken?

Java Swing is broken because it doesn’t use “native” components. Instead it paints all components you see in the user interface. However, a Tablet PC only provides the floating TIP button support for native Windows fields.

For similar reasons, Java Swing is not the only windowing system that’s broken. Firefox, a cross platform browser, is afflicted with a similar problem, I presume, because they’re not using standard Windows text edit controls. There is an add-on called GeckoTIP that adds floating TIP support for Firefox users on Tablet PCs.

Also, apps built on the cross platform GTK+ toolkit are also broken. I use GAIM (recently renamed Pigin IM) as an IM client. It uses GTK+ for it’s windowing. On my tablet, I have to use the non-floating TIP method to send messages.

So the problem is in the way that these windowing libraries are implemented. But trying to be cross platform and do their own rendering, they lose the benefit of the floating TIP on Windows Tablet PCs.

Why Hasn’t Sun Fixed Java?

Honestly, I don’t know. I’ll speculate that there are a few factors:

  1. The tablet PC community is still relatively small. It’s not gotten the large corporate evangelism that it’s members feel it should have. And frankly, using a keyboard-less tablet for general computing is a bit daunting to most computer users who have enough trouble making their software do what they want while using the familiar keyboard.
  2. Tablet PC users simply haven’t complained enough. If you search Sun’s Java bug database for “tablet pc” sorted by relevance it, seems that people haven’t filed all that many bugs regarding input methods issues specific to Tablet PC.s
  3. Sun doesn’t care about tablet PC users. For the bugs that were filed, there is been little or no response by Sun. As some of the non-Sun employee comments reveal, this isn’t necessarily about Tablet PCs. Some of the tablet input features are important for PDAs and smartphones sporting touch screens.

Regardless of why, history has shown that it’s not going to to get fixed anytime soon.

(FWIW, in my research, I found a comment here speculating that Java 6 was using off screen peers for all Swing text components and that might fix add native floating TIP support for Swing apps. But I experimented with the Java 6 betas and with the Java 1.6.0_01 release and it doesn’t magically add floating TIP support to Swing applications)

My Efforts

I recently started a new Java project that’s a Swing application that I expect to use on my UMPC a lot. So the lack of floating tip became very annoying and motivated me to pursue fixing the problem.

The “offscreen’ peer approach described in the comment referenced earlier gave me a good starting idea. I experimented with it and came up with the following.

The Approach

Basically:

  • Add a javax.swing.JTextField to the UI
  • Create a native java.awt.TextField and when the Swing field gets focus, add the native field as a sibling component to the Swing field, give the focus to the native field, and make it 1×1. This is enough to make it considered visible, but on Windows, the native field is actually not visible at this size. Doing this makes the floating TIP button appear near the native field.
  • Forward all keyboard events from the native text field to the Swing field so that what you type into the native field is reflected in the Swing field
  • As the state of the Swing field are updated (text, caret position, selection start/end), update the native field so that it’s state is in sync with the Swing field. This way if you do a copy or cut on it, it will contain the selection of the Swing field
  • As the caret moves in the Swing field, move the native field to the location of the caret so that the floating TIP button follows the caret
  • Disable the native field’s focus traversal keys. Instead intercept the keystrokes and check them against the Swing fields registered focus traversal keys. If encountered, transfer focus back to the Swing field, make the native field non-focusable, and request transfer of focus in the appropriate direction

It sounds intricate, but really isn’t all that bad. More importantly, it can all be done via event listeners. So there is no need to create special subclasses of the default Swing JTextComponents to modify the classes via code injection or some other form of bytecode modifciation. The real benefit is that you don’t have to change your UI to use differerent components that don’t behave like the default ones.

One other benefit is that it doesn’t use any native code. So it won’t have any special deployment needs.

Does it Work?

So far it does for me. And I asked for testing assistance over here at the Origami Project forums and a couple people reported that it worked and one reported that it didn’t.
Also I’ve encountered a few issues where passing special keystrokes such as Ctrl-Shift-End/Ctrl-Shift-Home doesn’t work. But I think I’m on the right track and can probably get it working more reliably.

Thus, I’m not confident enough today to say “I have a Solution!” But I am pretty confident that if I can find some testers to help me work out the kinks, that I will be able to reach that point.

Running the Tests

Do you want to help me test it out? I hope so. If you

  • Have a tablet PC
  • Are running Windows XP Tablet edition or Vista
  • Have Java 1.2 or higher installed in your web browser
  • And currently see the floating TIP button when you run native Windows apps

then run the test applet by opening this page.

What to expect

You should see an applet that looks like the following screen captures but without the frame and close button. There will be three fields.

1) The first is a native java.awt.TextField. When it receives focus, you should see the floating TIP button appear. A right click will bring up the Windows popup menu that contains Copy, Cut and Paste plus a few other possible menu items.

2) The second is a plain Swing javax.swing.JTextField. When it get’s focus, the floating TIP button will not appear, and to enter text without a keyboard, you’ll have to use the TIP from the task bar.

3) The third field is a Swing java.swing.JTextField that I augmented per the approach described above. If all goes well, when you put focus on it, the floating TIP button should appear nearby the caret. Try entering or editing text via the floating TIP.

I intentionally didn’t implement support for opening the Windows popup menu on this field because in most apps you probably don’t want a native menu appearing in your non-native look and feel driven application. At some point, I will implement a right-click context menu to provide Copy, Cut, and Paste actions. However, I will probably not implement specification of the input scope as this would definitely require a native call to Windows.

Give me Feedback, Please

Whether or not it works as expected, please let me know by submitting a comment below or but sending me a message.

If it works, I’d like to know, just to know.

If it doesn’t work, tell me more about what OS, Java implementation, and hardware you are using plus as detailed a description of what you are actually observing. It you try it, it doesn’t work, and you don’t tell me, then I can’t try to fix it.

Next Steps

If I can get some good feedback and work out the problems so that it reliably works, then I’ll pursue a generic method for floating-TIP enabling Swing applications…and do a Part 2 for this article. The approach will likely involve some sort of wrapping of the look and feel in use in order to get access to each JTextComponent as it couples itself with it’s UI.

If I can devise something reasonable, then I expect to publish the source and binaries so that the tablet community can, at least, explicitly floating-TIP enable their Swing apps.

I do acknowledge that after reaching that point, it’d be good to somehow transparently enable floating TIP for all Java Swing apps on tablets without having to change them them. I simply can’t imagine how to do this without resorting to native code or petitioning an RFE to the Java implementation. But I don’t want to get ahead of myself just yet.

This entry was posted in UMPC. Bookmark the permalink.

Leave a Reply