aboylikedave: Is CAT6 OK?
Given all this is so new to you, why not just go with a pair of Powerline adapters, as discussed earlier in this thread? Under $100 (the cheaper ones, at about $70, are slower, but would prob. do for your needs) and you're home and hosed, no need to worry about CAT6, what plugs to use, drilling holes, getting dirty under the house...
I extended my parents' wifi network on Saturday, connecting their new and old modem/routers (the latter as the 'extender' via a pair of Powerline adapters - and that's definitely something that could be done with a glass of wine in one's hand! Took only a few minutes, and the steps have been explained above (disabling DHCP, assigning static IP address, using same SSID....); the Powerline adapters were so easy to get working I thought I must be doing something wrong!
Often the biggest hassle is finding where in the router's settings to make the necessary changes, but even if not clear from the instruction manual a quick Google always helps.
(So, in summary - you're going to have to do the adjustments on the router either way, but the big hassle's going to be the cabling - that's where these adapters will be just so much easier.)