The DR-40 has balanced inputs; this means that the signal is carried on a pair of conductors and the shield is a separate conductor. The red and white phono jacks (commonly called "RCA" jacks) on your mixer are UNbalanced outputs; this means that the signal is carried on the inner conductor and the shield.
Consumer electronic gear almost always uses unbalanced inputs and outputs for the sake of low cost. Professional audio gear uses balanced inputs and outputs in order to guarantee freedom from electrical hum. The cable and connectors for professional gear costs a bit more, but they are inexpensive in comparison with the cost and frustration of tracking down and eliminating hum.
The input jacks of the DR-40 can accommodate either a 3-pin XLR connector for a microphone-level signal or a 1/4-inch TRS phone plug for a line-level signal. (TRS means tip-ring-sleeve; the tip and ring carry the signal, and the sleeve is the shield.) The EXTernal input switch on the left-hand side of the DR-40 chooses [LINE] or [MIC] or [MIC with phantom power]; the last is for condenser microphones.
The proper way to connect your mixer to the DR-40 is to use a pair of balanced (which is to say, two conductors plus shield) cables (one cable for the right channel and one cable for the left channel) with a 1/4-inch TRS phone plug on each end. Almost any mixer has a pair of balanced recording outputs with 1/4-inch TRS phone jacks. If your mixer does not have balanced recording outputs, get rid of it and purchase a decent mixer!
If you really wish to send an UNbalanced signal from the mixer to the DR-40, you have at least two options:
OPTION #1 :: Have a technician make a custom cable for you. The custom cable needs to terminate in a 1/4-inch TRS phone plug; for the wiring of the plug, refer your technician to
http://www.rane.com/note110.html. But be warned that you may end up with hum which cannot be eliminated. And note that the standard level for consumer gear is -10 dBV, which is about 12 dB lower that the +4 dBu standard level for professional gear; the DR-40 may not have sufficient gain for proper recording of a -10 dBV signal.
OPTION #2 :: Purchase an unbalanced-to-balanced transformer or converter box. You can find these, nicely packaged with connectors, at audio supply houses such as bswusa.com and fullcompass.com and sweetwater.com and markertek.com. One economical converter box is manufactured by an outfit called ART; it is the "CLEANBoxPro" two-channel bi-directional level converter. And though a transformer or converter generally takes care of the -10 dBV to +4 dBu level difference, neither guarantees freedom from hum.
Your local guitar or music store likely has in stock either a cable or a transformer or a converter box.
ANOTHER OPTION :: Purchase a DR-07 or DR-05, both of which have an unbalanced input jack. But unbalanced inputs and outputs are susceptible to hum; that is why professionals use balanced inputs and outputs. In once instance, a DR-07 was used to record from the unbalanced output of a professional telephone interface (JK THAT-2), but it was necessary to run the DR-07 on batteries, because hum was terrible when using an AC supply.
P.S. To switch between the internal microphones and the external microphone, you need to use the menus of the DR-40; press the MENU key.
