Catalysis of the transfer of rhamnose from one side of the membrane to the other. Rhamnose occurs commonly as a compound of plant glycosides, in polysaccharides of gums and mucilages, and in bacterial polysaccharides. It is also a component of some plant cell wall polysaccharides and frequently acts as the sugar components of flavonoids.