You could try some form of color quantization algorithm, which generally extract the most dominant N colors. The one I've seen referenced most is modified median cut quantization [.pdf], which is based on median cut quantization [.doc]. The benefit to this kind of algorithm is that instead of simply averaging every color in the image, it extracts and discards other highly prominent colors instead of allowing them to pollute the average.
The concept is that color space (RGB space) is partitioned into 3D axis-aligned rectangular regions (the paper calls them vboxes) and iteratively subdivided by splitting vboxes, attempting to leave half of the pixels on each side of the split. The result is color clusters that should correspond to color clusters in the image. The largest color has a strong likelihood of being "perceptually similar" to the image.
There's a JavaScript implementation and demo of this algorithm called Color Thief.