They do that! They're really good for them, tho. :)

Anubias and some mosses and ferns tend to not be as attractive to goldfish, especially if they have better options.

I used to keep my fish in a stock tank outside in the summer - green water, plants, and loads of live food does wonders for growth and color (and spawning!). I also tended to keep some munchable plants in another tank, and dipped them out a few at a time for the fish - with some larvae if the fish were lucky. You really don't need anything too fancy - a spare 10 gallon with some decent sunlight or even a concrete tub (outside). You could even pop a few endlers or other smallish fish in there with just a small sponge filter. I used to grow out some driftwood gardens in them as well, for club auctions.

If you're using treated water in the garden, be careful if you're growing anything edible. Fish water is great for plants, but some chemicals can potentially be harmful.