My opinion would be that the contacts inside your key switch that apply battery to the solenoid lug are possibly corroded or otherwise not making good contact.
You can be seeing 12 volts with no load (wire off the solenoid), but with the solenoid connected you will have current flow and bad contacts can represent a resistance to the current thus dropping your voltage.
I would remove the key switch and meter for good continuity between the bat and sol lugs while operating the switch.
