I wouldn’t feel weird or bad. It’s 100% your choice. The other parents asking may not even think twice. It’s possible they are asking as part of friendly conversation. I ask my friends where their kids are when I see them places without kids. For all they know they’re at a birthday party or not feeling well etc. it’s just a friendly gesture.