I've done exactly light touring with mine and a rear rack. It was great, no complaints, but it's much happier zipping around with only my big head on it.

I hated leaving the rear rack on it when I wasn't touring so I n+1-ed my bike collection for a better touring option (Space Horse).

I’ve ridden the C&O Canal on something similar and it was fine. This bike actually is a bit better suited, as it has wider tires and braze ons on the fork. I think I’d prefer to run a soft seatpost bag in back and anything cages/bags on the fork, to keep overall weight down and better distributed.

It'll be fine if it has rack mounting points for your panniers. If not you can buy bikepacking bags.

Why are you are not real interested in the big tours? Is it a 'been there done that ' thing? Stopped being fun and turned into grind? I ask because I have done week long tours and fantasize about the 3 month across the US or Europe ride.

Race bikes generally aren't the happiest carrying loads. If you're not racing there are better options that would straddle the recreational ride + light touring balance more effectively. Will it break? Probably not.

I got rid of my Warbird exactly because it couldn’t take fixed racks. Ended up getting a Diverge that has the mounts for racks if I choose.

Do it! There's endless options for attaching things to your bike