Can characteristics of skill sets confer positive and negative returns? To study this question, we introduce the concept of skill-set specialization, that is the average distance of the worker's skill set from skill profiles prevalent in the economy. We quantitatively show in a random search framework that more specialized workers i) suffer larger mismatch penalties on average across jobs, leading to lower job finding rates, but ii) enjoy higher gains from worker-firm complementarity in well-fitted jobs, reflected in higher starting wages and lower separation rates. Informed by the quantitative model, we analyze the labor market outcomes of exogenously displaced workers in the US and in France. We empirically confirm the findings of the model, thereby providing evidence for the two faces of worker specialization. The heterogeneity analysis suggests that specialization can have stronger adverse effects for lower skilled workers.