6 vs. 7 weight

I am currently using a TFO 8 wt. 9’6" rod and looking to pick up a lighter set up. I use the flyrod mostly while wading or kayaking but sometimes from the boat. I want something lighter as with most of the fish I am catching it feels like the 8wt. is a little much, kind of like king fishing with 50 wides. I was looking at a 6wt. but was told it was too small and should get a 7wt. Is there that much difference between a 7 and an 8? Is a 6 really too small? I feel like for every 36" bruiser red I am going to get taken to the bank by I will hook 50 smaller trout and redfish and would appreciate a little more sport in it. thanks for help.
cayman