I agree that, if you clean your shelves between smokes, the "seasoned" smoke residue on the sides, top and door, will have no effect on the taste of your meat. The inside of my smoker is pretty built-up, but my shelves are clean. That's what contacts the meat - not the sidewalls. With a clean shelf, the only influence on the taste will be a) your prep work on the meat, and b) the current wood you are using. Different woods have a distinct flavor profile, and the vessel does not effect that.
I will also add that wood smoke flavor is definitely very subjective. You need to learn what suits you, through experimentation. For example, I love hickory on just about everything, because that's what I grew up with in my area. Hickory smells like BBQ, to me. But, I also love cherry, mesquite, and several other woods. Apple? Hate it. Not sure why, but apple wood smoke flavor is very unappealing to me. Lots of folks love it, though. See what I mean? Some think hickory is "way to strong," but I don't.
Bottom line - use the woods that you like, clean your shelves, and don't worry about the other woods that have seasoned your smoker.