Immunotherapy using anti-PD(L)1 has revolutionized treatment for various tumor types. Early data have shown durable responses in a small subgroup of breast cancer patients. So far, the response rates appear higher for breast tumors that are triple negative, PDL1-positive and/or harbor high levels of immune cells. Both comprehensive analyses of the breast tumor microenvironment and exploiting research on biomarkers in other cancer types, such as melanoma and lung cancer, may contribute to the discovery of accurate biomarkers to select breast cancer patients for immunotherapy. Here we summarize key features of the breast tumor microenvironment as well as putative predictive biomarkers established in other tumor types. Insights from both fields can guide future studies to enable personalized breast cancer immunotherapy.