Youre quite right: bits are bits, and the "signedness" or "unsignedness" of a given bit pattern depends strictly on your point of view.
Wikipedia foundation.