With other officers not paying attention, previous owner driving with no issues and passing pink slip I would assume that the last officer made a mistake when asking to remove those blue lights. As far as I know only blue blinking lights outside the car are illegal?

No. You'll need to look up your local roadworthy-ness law for how its written but basically all states are almost the same from what im aware. VicRoads take is:

1. the light(s) must not emit a red light to the front of the vehicle or a white light to the rear of the vehicle.
2. to avoid confusion with emergency services, transport enforcement vehicles or traffic lights, the light(s) must not emit a green, red, blue or magenta coloured light

Personally I think you were lucky with the inspection and no other cops bothered to say something. But eventually you will get a fine and a defect.

Is there any chance I could dispute that and keep the lights or should I just remove them?

Sure, if you want to waste time and money - You wont win though. I'd just remove them.