The pertinent specifications for using headphones with the DR-40 are:
Output impedance: 12Ω
Maximum output: 20 mW + 20 mW (when connected to headphones, 32Ω load)
I would interpret this to mean that the DR-40 is
capable of driving headphones of 32Ω at full power output of 20mW in each channel.
It sounds like the DR-40 might be designed to be similar in output to portable music players (iPods, etc.), which often are optimized for use with 32Ω headphones.
Lower impedance 'phones (less than 32Ω) would probably require more output current than the output stage is capable of producing, causing overload and increased distortion. I don't know of any good "modern" headphones that are less than 32Ω, anyway.
Higher impedance 'phones would use less current, but would require more voltage for full volume. Portable players can't provide much output voltage, so they often come with 32Ω 'phones, and supply more current for higher wattage (volume).
This morning I tried my DR-40 with four different sets of headphones. I tried a set of older Apple earbuds (32Ω)
Sennheiser HD 280 Pro (64Ω)
Sennheiser HD 600 (300Ω) and
Sennheiser HD 420 (600Ω).
Maybe I'm lucky (and I
don't like to blow my ears out), but I got decent clean output using all four 'phones. The output level got uncomfortably loud with all of them before I heard obvious distortion. The Apple earbuds sounded the worst (very bright and tinny), but I don't know if that is due to their low impedance or their "quality"

.
I got good volume levels at around "halfway" on the volume indicator with the 32Ω and 64Ω 'phones, and about 3/4 level with the 300Ω 'phones. The 600Ω 'phones were JUST loud enough at full volume.
I looked at the output waveform on my oscilloscope (I was powering the DR-40 with three fresh Duracell alkaline batteries).
The output voltage clipped at about 3 volts peak-to-peak with the 32Ω and 64Ω 'phones, and at just under 4 volts p-p with both the 300Ω and 600Ω 'phones.
All but the 600Ω 'phones were
far too loud to keep on my ears at clipping level. The 600Ω Sennheiser HD 420s were tolerable at full volume, which was right at clipping.
So, if you're hearing excessive distortion from your DR-40 I would try using 'phones of at least 64Ω, and preferably higher, up to 300Ω or so). The DR-40 just can't adequately drive 600Ω 'phones.
And make sure you're using fresh batteries.
Of course, you will get MUCH better sound if you plug your DR-40 into a mixer or headphone amp, and plug the phones into that!