« June 2003 | Main | August 2003 »

July 25, 2003

My Very First USB Peripheral

prototype hardware

Earlier this month I talked about the Microchip PIC16C745 and PIC16C765, which are 8-bit microcontrollers with built-in low-speed USB. Since that time, I've got hold of one of these devices and built a simple USB peripheral with it. Click on the thumbnail for a bigger version of the image.

Summary: hardware easy, firmware tricky, software painful. Read on for more detail and some hints for anyone treading the same road in the future.

[Last updated 20050606: added firmware source code.]

Hardware

As you can see from the image, the hardware is simplicitly itself. (If you're really interested, you can look at a schematic). You can see the USB B connector at the far left, carefully soldered to some wires: I did take care to continue the data connections as a twisted pair, but other than that it's really not subtle.

The PIC16C745 is in the centre; you can see that this is the ceramic /JW package from the little quartz window over the die. You uncover this when you want to reprogram the chip; it takes about ten minutes cooking in my UV eraser. Theoretically, the processor can start up badly unless you cover the window when the chip is in use, so I usually put a little orange sticky dot over it on a programmed device.

Above the processor you can see a crystal and two capacitors giving us the required 6MHz reference frequency. It turns out that low-speed USB is more flexible about frequency accuracy than the faster variants and that a ceramic resonator would have been sufficient and a little cheaper. That's probably more a concern for people interested in production quantities; you can get a bag of those things for a fiver.

Just to the right of the processor you may be able to make out a decoupling capacitor and a resistor holding the chip's reset pin high; just to the left of it you can see a resistor and capacitor that allow the device to pull up one of the USB signals at startup to indicate that it is a low-speed device. The bit of black sticky tape is there to remind me where the end of the chip goes after reprogramming it.

This is everything you need to get the device up and running; the remaining components on the breadboard are four little switches, each with an in-line resistor to help protect against static discharge, and two LEDs each with a current limiting resistor.

Firmware

Things started well on the firmware front. Microchip do have standard firmware packages for these devices linked directly, for example, to the PIC16C745's index page. Even better, though, they have a series of "canned applications" that are more useful for tutorial purposes, provided as technical briefs. The only problem is that these are almost impossible to find on their web site. For reference, I found all of them by reading through the list of all application notes and picking up everything that sounded interesting.

I started by downloading Technical Brief 54 and its associated source code package. This is An Introduction to USB Descriptors - With a Gameport to USB Gamepad Translator Example and the ZIP file contains a hex format file you can burn directly to a PIC16C745 or PIC16765. Plug in the USB connector and the device should be almost instantly recognised as a gaming device: no other drivers are required, at least on Windows since the 98SE version and under Linux 2.4. You can play with your new USB "gamepad" in the Windows control panel.

This surprisingly handy effect comes about because the firmware identifies the USB device as a HID class device: HID standing for Human Interface Device, i.e., one of those terribly slow things whose data rate is limited by how fast humans can twitch. Within the HID class, there is an additional specification to allow a device to enumerate its controls, their values and the usages intended: for example, a joystick device would have a couple of linear axes and some on/off buttons.

The problem comes if you want to deviate from someone else's working setup, as you almost certainly will at some point. For example, I wanted to change the number of buttons and add a couple of LEDs. To do this, you need to build new HID report descriptors and there you run into problems with incomprehensible standards documentation and inconsistent operating systems implementations.

There are several different kinds of descriptor you need to build to make a working USB device. Some of these are described in the USB standard itself (hint: for most purposes, just read the 1.1 standard as it is substantially shorter), some in the HID adjuncts to that standard. The most difficult of all to understand are the HID report descriptors and the system of usages and usage pages: I found that apparently sensible changes to a descriptor would just silently "not work", resulting in a device that was invisible to Windows 2000. My advice is to make small changes, step by step, from a known working system to where you want to go. Back up at every stage before making a change, and revert immediately if something doesn't work. Don't be tempted to make too many descriptor changes at a time, even if it does take ten minutes to erase the device each time you want to make a change. Under no circumstances attempt to build descriptors from scratch, as most likely your device simply won't configure; you get essentially no information as to the reason for this. If you'd like to compare the TB054 source code with my modified version, you can download my firmware source code.

There is definitely an operating system consistency issue as well: for example, I have seen configurations of controls that are acceptable in Windows 2000 but cause Windows 98SE to reject the device. You need to be able to test your firmware against all the target systems in order to be sure that it will actually work there.

The USB HID page includes a tool to be used to create HID report descriptors; it can perform minimal checking on the ones it creates or be used to do the same minimal checking on ones you've constructed elsewhere. It's not infallible, though.

The other tool provided by the USB Implementers Forum that looks like it should be useful is USBCV. This only works under Windows 2000 and XP, and only works in systems with both a high speed (USB 2.0) host controller and high speed hub, even for testing low speed devices. Unfortunately, I don't have a system meeting all of those requirements at present so I can't yet say how good a diagnostic tool it is.

Other resources I found of use while debugging firmware:

One last item that fits in under "firmware" better than anywhere else is the issue of vendor IDs. Like PCI and other "plug and pray" bus systems, your device presents a vendor ID and a product ID so that the host operating system can locate appropriate drivers. Unlike those other systems, there is no "experimental" vendor ID or apparently any way to get a legitimate vendor ID other than paying the USB Implementers Forum a minimum of $1500 every couple of years. Microchip do have a vendor ID, but as far as I know they don't have a programme in place to hand out product IDs to customers as was the case for many PCI chip vendors. If you do the obvious thing and pick two random 16-bit numbers or some defunct company's vendor ID, everything will (probably) work but you can't use the USB logos on your products.

Software

Host application software for a custom peripheral is where things get really painful. This isn't the case if you're building a joystick or a mouse or some other device for which operating systems already provide all the drivers that you need; all you need to do is fire up Flight Simulator and you're done. If you want a host application of your own talking to a collection of custom peripherals you've designed, though, things are much harder.

Ideally, I'd want to be able to write a client application in a nice safe language like Java and have some hope of it talking to my USB devices under at least some variants of Windows and Linux. The bad news for me is that there is no stable Java API for USB yet, and neither of the attempts to date appear to have been ported to Windows.

The older of the two extant APIs is Java USB, which appears moribund and is currently Linux-only. Some Windows code was submitted to the project a year or so back, but it doesn't appear to have been developed or available for use. The newer API is javax.usb, which apart from being quite active is also the subject of Java Community Process JSR-80: in other words, the aim is to make it into a semi-official standard. Unfortunately, this project is also currently Linux-only; something in the way of a Windows port is anticipated around the end of 2003 or early 2004.

One option I have considered is to make use of the fact that if I continue to make my peripheral pretend to be a gaming device of some kind (with some fake axes that never move), getting data from it can be done through Microsoft's DirectInput API. It just so happens that an enterprising German developer has something that allows just this kind of operation from Java under Windows: this is JXInput; it works by using a JNI (Java Native Interface) library to bridge across from the Java VM to the DirectInput objects. This is a promising avenue, but unfortunately DirectInput gives essentially no way to write data back to the device, apart from the "force feedback" facilities. JXInput does not even support the force feedback option: this means I could read my buttons but couldn't light my LEDs.

Microsoft's DirectInput documentation covers this eventuality neatly by suggesting that the Windows HID functionality be used directly for these purposes. Of course, this could be the route for both data directions and another option would be to write my own JNI wrapper for the appropriate Windows API calls.

Stepping back from Java, there are of course several alternative programming environments for Windows. I know both C++ and Visual Basic well, and example code of varying degrees of crudity are available for these languages in various places (start at Jan Axelson's site's USB code page). However, neither language is really as good with multi-threading or asynchrony as Java is, and I really dislike the idea of learning to program the Windows GUI in C++ at this stage, having avoided it for so many years. Microsoft's C# would be the new trendy alternative, but I'm unfamiliar with it except as being "Java like" at some level and having got the "Hello, C# world" console application running.

Curiously, one language for which USB HID support does seem to be well done under Windows is Delphi, about which I know even less than I do about C#.

You can see from this that by far the most difficult part of the whole enterprise is software for the host. At present, I'm using an adapted version of some of Jan Axelson's Visual Basic code but I'm far from happy that I could scale this up to drive several similar devices in an asynchronous way, and unfortunately that's what I have in mind when I get beyond the pure experimental stage.

If I'm lucky, JSR-80 will happen before I get there.

Posted by Ian at 10:26 PM in Hardware | Comments (41) | Permalink

July 6, 2003

Order of the Phoenix: Unanswered Questions

Please sit down.
I am going to tell you everything.
   — Albus Dumbledore, to Harry Potter

Well, Dumbledore may have finally come clean and answered all of Harry's questions, but from a technical perspective I feel J. K. Rowling still has some explaining to do. Here are some areas she might consider addressing in the next volume.

Eight Oh Two Point What?

There's a disturbing trend in modern fantasy for the women to be the natural geeks (as opposed to the natural nerds, who are still male, of course). How long will it be before Hermione's parents give her a laptop for Christmas instead of a year's supply of dental gum?

More to the point, given that Goblet of Fire established that electronics won't work in the school grounds, how long will it be before she starts trying to get someone to install a WiFi hotspot in the Three Broomsticks?

If Hogsmeade does get this kind of amenity, I can't see that a Starbucks could be far behind. I'm sure a school full of wizard kids high on grande lattes would mean a world of trouble, far worse than that "butterbeer" they're guzzling this year. (Incidentally, how much alcohol is there in one of those? Isn't there a law or something?) Apart from the discipline problems brought on by the hyper-activity, you have to take into account the odd spell going astray due to a shaky wand hand. Nasty.

Handy for Jump Starting Your Car, Too

Speaking of electronics, why do electronic systems fail in the Hogwarts grounds? Is it some kind of electro-magnetic pulse (EMP) effect? If it is electro-magnetic, can the energy be harnessed by normal humans, Matrix-like, by crocodile-clipping a couple of sturdy cables onto a wizard's ears?

Perhaps this is the real reason wizarding folk make sure they are never seen by muggles...

Taxonomy of Magical Creatures

Somewhere in the school library, Hermione will one day come across a book describing the ways in which magical creatures are related to each other, and to non-magical creatures. I'd like to request that this scene be included in the next installment if possible. It's not as if we're likely to be short of space, volume 5 being 766 pages after all...

By the way, I don't mean books like Newt Scamander's Fantastic Beasts and Where to Find Them: I've read that, and it's purely descriptive natural history. You'd think Darwin had never been born as far as the wizarding folk were concerned; this despite his rather wizardly facial hair.

What I'm really looking for is a book that will tell me whether Unicorns and Thestrals are descended from "normal" horses or the other way around. Alternatively, perhaps we're looking at a case of convergent evolution. Is there a fossil record we can use to help us out with this kind of question? Indeed, are the highly unconventional body plans seen in the fossils of the Burgess Shale something the Ministry of Magic would have preferred to have covered up?

As a bonus, the Latin binomials for things like blast-ended skrewts would surely be of use to the serious student.

And talking of books, does Hogwarts, a History have an ISBN? I can't seem to find this book on Amazon.

And Finally...

We're never told how the magic in Harry's world works, even in general terms. This is a serious omission, in my view.

It's fairly clear that we're not looking at a mana-based magical system like Niven's, as if we were then areas like Hogwarts and Diagon Alley would be magical wastelands by now.

Perhaps it's a "true names" based magic, as seen in Le Guin's Earthsea novels? It's not clear whether wingardium leviosa and the like fit into a scheme like that. Maybe its all done behind the scenes by some relatives of the house elves.

Inquiring Minds Want to Know

Ian A. Young, aged 43 and a quarter.

Posted by Ian at 11:04 AM in Books | Permalink

July 5, 2003

Shirky: A Group Is Its Own Worst Enemy

Clay Shirky writes about the Internet, as much as a social and cultural phenomenon as a technical one. His recent article A Group Is Its Own Worst Enemy describes what happens when "social" software (Usenet, mail, instant messaging, weblogs, etc.) is used to support growing, long-lived user groups. In particular, he discusses patterns of failure in these groups and to a lesser extent what can be done to avoid group failures.

It's far from a recent observation that members of a group can cause that group to fail because their motivations aren't the same as those of the group as a whole: I think I personally first saw this kind of effect on Usenet in the early 80s. In fact, this is one of Shirky's main points: people persist in setting up groups mediated by software without learning from the failures (and successes) of previous attempts. Then, of course, they are surprised by the results and write another paper which is in its turn ignored by the next generation of designers...

One of the examples discussed is a well-intentioned but unstructured 1970s Californian bulletin-board system subverted by an inflow of adolescent schoolboys. Some of the most interesting material described comes from pre-Internet psychological work; the point being that failures happen because the motivations of the members of a group can cause problems when they conflict with the purpose of the group, and that this is a human social issue rather than a technological one.

On-line (or partially on-line) social structures are worth trying to protect from the kinds of failure pattern Shirky describes. Unsurprisingly, he can't give us any instant cures for these ills; it may be that there are none. Instead, he supplies some necessary but not sufficient conditions for a functional long-term community:

This is a long article, but it's well worth the time. One thing it could have done with was a link to the LambdaMOO messages he references, so here is one.

Found through an article by Daniel H. Steinberg at java.net.

Posted by Ian at 6:05 PM in Miscellanea | Permalink

July 3, 2003

PIC16C745/765: microcontrollers with USB

Fun devices of the month: Microchip's PIC16C745 and PIC16C765. As well as the usual microcontroller features, these require only a couple of external components to function as low-speed USB peripherals.

It is getting very easy to put together small but sophisticated electronic doodads using microcontrollers. Twelve years ago, I built some custom hardware using a 4-bit National Semiconductor COP401 microcontroller: it was simpler than the Z80-based systems I had built in the 80s, but it still involved a separate EEPROM for the code and I had to write my own assembler in order to code for the device.

Recently, I've been looking into building a more modern version of that old COP-based system, and potentially upgrading the functionality so that the new system can be hosted by a standard PC rather than having its own display built in. Rather than get into parallel port interfacing (so last century) I'm considering making the new system a USB device.

My current best finds for this function are two microcontrollers from Microchip, the PIC16C745 and PIC16C765. The smaller PIC16C745 even comes in an easy-to-use skinny DIP package so I don't need to worry about getting into surface mount manufacturing.

Just to drive the point home, here is what you get in a PIC16C745:

All of the above comes to you for a cost of less than £5.00 in singles; much less if you can buy even a couple of dozen at a time.

You can see that a device like this looks like it would make designing a simple HID (Human Interface Device) class peripheral like a control box or a display very simple. I might write up some more details if and when I manage to do that.

I do have two minor quibbles about these devices, which probably result from their coming onto the market a couple of years ago and therefore not using Microchip's latest technology. The first is that the devices use EPROM memory for code rather than FLASH as with most of Microchip's other current parts. This means that the plastic-packaged production units are one-time programmable rather than reprogrammable, and also that development has to be done with a "windowed" ceramic package device, which you re-use by erasing using a high-intensity UV lamp. Fortunately, I still have one of those left over from from my mid-80s Z80 days...

The second quibble is probably related to the first in terms of the underlying technology: most other current Microchip parts have a small chunk of EEPROM you can use to save things over power cycles; these parts don't. That's wouldn't be a big deal for my intended application were it not for the possibility of using EEPROM to ameliorate the "program once" nature of the devices.

Even with the quibbles, these chips look like they'll be really useful.

[Later post on this subject: My Very First USB Peripheral.]

[Last updated: 22-Aug-2003.]

Posted by Ian at 5:10 PM in Hardware | Comments (11) | Permalink