Heres the symptoms:

In neutral, I can hear a whirring/whine noise pretty distinctly that goes away as soon as I push the clutch in. No noise with the clutch pushed in, and a whirr when the clutch in engaged.

Is this something I need to worry about?

If it is the throwout bearing, is this a sign that my clutch is beginning to fail? I've been pretty rough on it a few times offroad, and I tow a lot.

 

It sounds like it is not the TO bearing, or the pilot, or the clutch at all. Sounds more like a tranny bearing. Any others have an opinion?

 

Sounds identical to my problem. It turned out it was the output shaft bearing($70 at the local stealership)
I put a new clutch kit anyways and also a good idea is to pick up a new seal when removing the shaft bearing. You may or may not need it but it's only $8 and will save you a couple of days if yours gets damaged.
The bearing was a bitch to remove and we had to use heat and burn it out. Scored my gears in the process and ended up buying a used tranny in the end. Go SLOWWW man and you should be allright
Good luck

 

I agree that it sounds like tranny bearings. But, many transmissions make a bit of noise when in neutral with the clutch out. If it makes noise when moving in gear then there is definitely a problem. Take it by a couple of tranny shops and let them have a listen.

 

My dad's 2002 Tacoma makes a high pitch whining or whirring noise when the clutch is out and in neutral; as far as I know it's done it since new. If that's what you are referring to, I don't think you have a problem.

 

Sounds like the TO or Pilot bearing to me.

 

Well I was finally able to hear the noise even when the tranny was in gear.

What does this mean? Transmission bearings?