Hi and Welcome!
Being a mechanical clutch, it's completely dependent on the radiant heat from the engine to engage or disengage. Quite frankly, it's not accurate at all... at least not compared to an electrical fan with a thermostatic switch. Mine does the exact same thing that you're describing; roar on startup that disappears after 10-30 seconds. Not always there in traffic, but sometimes I can hear it whistling away.
There are a number of write-ups on electric fan conversions if it bothers you. That would the the 100% solution. Replacement fan clutches are listing at $70 at rockauto.com via eBay if you think it might just be the clutch assembly.
Good luck!
PS: The link for the clutch: Click Here!!
__________________
•-Noltz-•
'91 4Runner SR5, 3" BL, 35's & 8K Winch
'97 4Runner Limited, Stock w/E-Locker & 285s, DD
'95 4Runner SR5, Bone Stock, DD