I'm probably not the best guy to recommend where to buy Japanese music—99% of my CDs are purchased at the Tower Records in Shibuya... But there seems to be some interest in this subject, so...
First, Tower Records. Unlike in other countries, where Tower is said to be nothing but a Faceless Corporate Chain (...or, was one—it went bankrupt, right?), the stores in Shibuya and Shinjuku are huge but have a human feel. A big part of the appeal is that the Tower staff hand-write CD recommendations on cards and stick them onto the store shelves—those recommendations are helpful, and it also shows that the place is run by people who care about, and even love, the music that they're selling. (Though they were better about this, I think, before they remodeled and got rid of their Japanese indie section, which I wrote about here...)
In Shibuya there's an HMV store down the street from Tower, another giant store, but I don't shop there much because it has a smaller selection of Japanese music, although I hear it's a good place to buy vinyl.
Shibuya, Shinjuku and other parts of Tokyo also are home to a zillion small record stores. In Shibuya there are lots of little stores specializing in hip-hop, house and club music, many of them in the Udagawa neighborhood (sung about in “Udagawa Friday” in case you're a Capsule fan). It's also home to the cool little indie pop store Apple Crumble Record. Shinjuku, meanwhile, has a number of punk record stores. Here's an article on Tokyo record stores in the 'pop culture travel guide site Jaunted.
Online music shopping-wise, I use Amazon Japan a lot. You can make it display in English by clicking on the 'IN ENGLISH' button on the top right, but the only problem is that it doesn't translate artist and album names that are in Japanese, so if you can't read Japanese it will be nearly impossible to find anything.
For example, let's say you wanted to buy Orange Plankton's classic album from 2003, Mizu No Niwa [Garden of Water]. If you search for 'Orange Plankton' or 'Mizu No Niwa', Amazon will blow you off by saying your search “didn't match any products”. You have to search the terms in Japanese to get what you are looking for. The same goes for all the many Japanese artists and bands that go by a Japanese rather than alphabetical name.
An alternative is a site called CDJapan, though I've never bought anything from it myself. A quick series of searches revealed that you can find, in English, favorite artists of mine such as Asakusa Jinta, Tornado Tatsumaki and Yuyake Lamp—none of which will come up in Amazon Japan via English searches. I'm impressed by their inventory.
You can also buy Japanese music MP3's online. The best site for this that I know is JapanFiles.com. They have a great collection of artists, including personal favorites such as: advantage Lucy, Asakusa Jinta, Luminous Orange, Mix Market, Macdonald Duck Eclair and Swinging Popsicle.
Another MP3 website I've heard about is called HearJapan. It's new but looks promising—I think the challenge for them now is to get a critical mass of artists, so that they have something for everyone, or, at least, everyone that likes indie Japanese music (personally, though, I'd prefer it if you didn't have to log in at the start, but only when you actually decide to buy something. Being able to browse freely is a good thing). Harvey of JapanNewbie.com writes about HearJapan here.
Finally, if you want to listen to some music for free, music that's completely oriented toward MY tastes, you can always tune in to Japan Live Radio, which I updated with new Chara, Mix Market, Coltemonikha, and other stuff. I've made 27 song streams already since I started the radio—it's a Labor of Love.
Let me know if I missed anything, or you know other good stores.
UPDATE: Commenter smashingtofu recommends YesAsia.com, which is said to be "very reasonable in shipping". I also like it that they have Korean and Chinese music in addition to Japanese stuff.
Johan Nystrom praises tokyorecohan.com, saying it's great for buying used CDs. Tokyo Recohan is chipple.net writer Patrick's project.
Also, I should mention the article "Record Shopping in Japan" in the super-cool site TweeNet.
Japanese Mystery File, Entry #385: This train station poster is to tell people that if they want to smoke, they need to go to the Smoking Section (yes, in non-puritanical Japan, you can still smoke outside...). But...why is the featured character a Cigarette-Smoking Bamboo Shoot??