When they are in a commercial food, I'd say you are right to avoid them. From dogaware.com:

By-products may be OK if the company specifies that they are human-grade organs such as liver and kidney, but otherwise they usually signify parts not considered fit for human consumption.
So, that's just the point I was making. I feed what are considered by products (organs, etc) in a raw diet, but you really don't want generic "by-products" in your dog food.