When someone says “rare operating system”, the word “rare” describes “operating system”.

Here is the statement again:

Normal people either can’t afford these devices or don’t have time for all the hassle of installing and using a rare operating system on a phone.

In your sentence, “rare” is used to describe “it”, a pronoun, which refers to the action “to walk around with an actual tinfoil hat”.