The problem was with the lower bolt on the caliper. As illustrated below, with the ratchet attached.

I can loosen it to the point where it seems like it should just come out but it wont. It takes a lot of effort to keep turning it, it's like it's come to the end of the thread but as if it's clamped in rubber, but it doesn't come out any further.

Am I missing something here?