I don't mean damage caused to the socket and prongs, but only to the appliance itself.
Another example of needing to be careful is in situations where heat removal is involved (such as internally driven forced-air cooling of a video projector bulb). In these cases, to prevent short-term (or longer-term) damage, you need to wait until the device has had a chance to properly cool itself down internally.
Typically, an appliance will not be damaged if it is currently ON and then you unplug its power cord. If you plug it back in the device will simply resume operation as if it were ON.
I say typically because there are some cases where this may not be the case. There are many devices such as computers and some other electronic devices that almost always recommend that you interact with the device to gracefully shut it down before you think of yanking the power cord out of the wall. This graceful shutdown process ensures that the device has finished all critical internal operations before losing power. This could include operations such as closing open files or saving important system state to a storage memory.
With many of today's appliances having more and more smart electronics, it can be more of a concern that you not arbitrarily pull power cords. A well-designed appliance from a reputable manufacturer will have made provisions in the electronics design and software so that the device does not get bricked when the power is suddenly lost. On the other hand, as more designs are done by less experienced people, this possibility of damaging operation of an appliance by the sudden loss of power is more likely than ever.
To make my case, let me comment that the ever more popular Raspberry Pi (rPi) microcontroller board is leading many people to think of embedding devices like that into appliances and other gadgets. It is a known fact that removing power from an rPi board can brick the boot media on its micro SD card and keeping it from being able to reboot when power is restored.
tl,dr; it is probably OK for very low power devices. For high power devices you can accelerate wearing on both the plug and the wall socket.
In most devices the power switch does the same thing you do when you pull the plug from the wall: it interrupts the circuit. There are other devices that are always "on", and the power switch just puts it into hibernation mode.
For the second kind there is a bit of electric noise when you unplug it but modern, well designed electronic circuits can withstand this (at least within their nominal lifetime). As some people have mentioned, some sophisticated devices have computers inside, and some of them are plugged into the Internet. Although the physical device itself is not likely to be damaged, if the device is receiving a software update through the Internet and you unplug it before the upgrade is completed you can "brick" it (it will not start because the software will be incomplete). If your device does not receive "OVA" (over the air) updates, you should be safe - of course do not unplug it if the screen says something like "Please do not unplug until the upgrade is completed".
Simpler devices like a heater or a vacuum cleaner will not be damaged, but depending on the currents involved doing this often can accelerate wearing of both the plug and the wall socket. For higher currents the switches are designed using "arc suppression" techniques in order to withstand the stress of cutting the circuit. Look at this picture of pristine contacts from a brand new switch and the same contacts after 100,000 on/off cycles:
Where the damage comes from? When you cut the circuit, the electrons will try to keep flowing and will fly through the air causing an "electric arc" (if the room is dark enough you will see the spark). The temperature of the resulting electric arc is very high (tens of thousands of degrees), causing the metal on the contact surfaces to melt, pool and migrate with the current. The high temperature of the arc cracks the surrounding gas molecules creating ozone, carbon monoxide, and other compounds. The arc energy slowly destroys the contact metal, causing some material to escape into the air as fine particulate matter. This very activity causes the material in the contacts to degrade quickly, resulting in device failure. Often the switch contacts are coated with some noble metal like gold or silver.
Now look at your wall plug: it is not designed to suppress the electric arc, and it is not coated in a special material. It will fail way sooner than a well designed switch. Because of course wall plugs are not designed to be used as an electric circuit switch!
How dangerous is that? A good quality wall plug can withstand a lot of abuse but once the contacts lose the outer protective layer they can decay very fast causing the plug to heat making plastic parts slowly burn and become conductive eventually causing a short circuit. The circuit breaker should be triggered by the excess current and shut down before the house is on fire so under ideal conditions it is not that dangerous. That said, my advice is to avoid pulling the plug with the appliance on for anything above 1200 w (10 ampere at 120 v).
There are outdated (or plainly illegal) electric wiring, old/defective circuit breakers, cheap low quality plugs and so on. If your plug is humming, flickering, heating or smelling smoke replace it immediately.
Source: I'm a certified industrial electrician and electronic circuit designer.
Some electronic devices with on-off switches can be damaged when being plugged in. The arcing can damage static sensitive devices. Many electronically controlled devices have debounce circuits because even pressing a switch causes spikes, but if plugged in with the switch on the spikes are many times higher as the electronics power up so at least when plugging things in, it will reduce the possibility of damage if the power switch is turned off.
Practically none will be damaged by an unexpected power failure. Customers get cranky about that sort of thing.
Anything software driven with non-volatile storage may have difficulties if it was either in the middle of some critical operation (rare) or you frob the power on and off multiple times without giving it a chance to repair its file system.
A bunch may require some judiciousness or care when plugging them back in. For example: Dishwashers and washing machines will still have water in them, and may require some poking at the controls to convince them to drain rather than attempt to fill a second time and overflow. Also I happen to have a freezer on which the compressor motor is not strong enough to start if there's already pressure in the system and it will blow a fuse if you try, so if it's interrupted mid-cycle you have to let it sit for ten minutes or so to allow the refrigerant pressure to bleed down before you restart it.
It depends on the device.
To take Michael's example of the Raspberry Pi, Flash memory (which works like RAM) is destructive on reads, so to read data you have read it out and then write it back. If you pull the power when it's in use you might be mid-cycle and lose data. Many SSD(flash) drives for more expensive computers have a capacitor that keeps the power on just long enough to complete the cycle if the power goes out.
Appliances don't generally have this issue because they tend to use ROM chips instead. A power failure doesn't matter because the data on the chips isn't volatile. If they are using Flash memory anywhere, it's probably not for mission-critical parts (an appliance is something you're likely to repair and warranty repairs are expensive to companies).
Aside from some wear on the plug and socket that you specifically mentioned that you aren't asking about, there is no harm in pulling the plug on something that is running unless it's a Windows 7 or later computer that is running Windows Update.
Everything is made to handle an unexpected power outage unless it's from Microsoft. One rare exception might be something like a machine tool that has a small light bulb and a large motor that is controlled by a switch. Pulling the plug while the switch is still on and the motor is still running under load may cause an inductive spike created by the motor to get absorbed by the light bulb and burn it out. This situation also applies to any large inductive load that is connected in parallel with something more fragile. When a circuit breaker or switch disconnects everything together the inductive spike created by the inductive load will be exposed to the more fragile device, without the original power source connected to absorb the current. This is a common situation and pretty much everything is designed to handle these types of Voltage spikes, as long as the size of the inductive load is within reason. You'll often find MOVs (surge suppressors) built in to power supplies.
A more interesting version of your question might be: What happens if I unplug a power strip with a variety of appliances connected to it, such as a large operating non inverter type microwave oven and a cheap imported USB power supply?
Another issue not yet mentioned is that some devices have series capacitors without bleeder resistors. In some cases, if such a device is unplugged without switching it off first, an internal capacitor connected to the AC line might remain charged for awhile, creating two related hazards. First of all, touching the exposed prongs of the AC plug when the capacitor is charged may deliver a rather nasty shock. Secondly, if the line phase when the device is next plugged in doesn't match the line phase when it was disconnected, it may cause a nasty current spike the next time the device is plugged in.
Properly-designed devices which use series capacitors of any significant size should include bleeder resistors to ensure that any stored energy is dissipated within a few seconds at most. Bleeder resistors would waste power continuously while a device is plugged in, and a bleeder resistor which discharges 63% of the stored energy in one second would waste twice as much power as one that would take two seconds, so there are advantages to having things discharge somewhat slowly. Thus, even if a device includes a bleeder resistor, unplugging the device and immediately touching the plug could yield a nasty shock. Generally, however, such a shock hazard would only exist while the switch is in the "on" position. Turning off the device before unplugging it would make the plug safe to touch, at least as long as the switch remains off (turning on the switch sufficiently soon after unplugging the device could cause the plug to yield a nasty shock).
Note also that while any safely-designed device that stores significant energy should include a bleeder resistor, some cheap import devices that aren't subjected to any kind of regulatory inspection or testing might save a fraction of a penny by omitting the bleed resistor. Touching the exposed prongs of such a device could yield a nasty shock even if it's been unplugged for hours.
The answers so far state computers as an example; but they're very poor examples. The damage done to the computer hardware is nothing if you rip the power cord out. They don't run at high currents, and the PSU tends to be very well designed to ensure that the output voltage is sane.
However data caches will not be flushed, and data consistency may not be promised. You could end up with a perfectly good computer - which needs a re-installed OS; and again with the pi example mentioned in an answer; the hardware itself is fine.
However, if you have something that has 100A+ running through it, and you kill the power - the hardware itself can be destroyed due to the very high amount of stored energy in anything with inductance that will suddenly be going in the wrong direction. It's not that you can't protect against this; but that the cost of doing so can be prohibitive.
The things that will have this much current though them however will tend to be something you won't have in your house; but in an industrial setting.