Can specialization – the average distance of the worker’s skill set from skill profiles prevalent in the economy – confer both positive and negative returns? We quantitatively show in a random search framework that more specialized workers i) suffer larger mismatch penalties on average, leading to lower job finding rates, but ii) enjoy higher gains from worker-firm complementarity in well-fitted jobs, reflected in higher starting wages and lower separation rates. The theoretical model indicates that the acquisition of more specialized skills may or may not yield an augmented lifetime income for workers. Hence, the acquisition of more specialized skills can come at a loss of lifetime income for some workers. Informed by the model, we analyze the labor market outcomes of exogenously displaced workers in the US and in France and provide empirical evidence for the two faces of worker specialization.