Natural Language Processing technologies have advanced drastically in recent years, and they have been applied in various real-world applications that touch our daily lives. Despite their remarkable performance, recent studies have shown that these technologies are vulnerable to adversarial examples, make predictions based on spurious correlations, and run the risk of aggravating the societal biases present in the data. Without properly quantifying and reducing the reliance on such correlations, the broad adoption of these models might have the undesirable effect of magnifying prejudice or harmful implicit biases that rely on sensitive demographic attributes. In this talk, I will take governing societal bias as a running example in order to demonstrate a collection of results that lead to greater control of NLP systems, such that they are socially responsible and accountable. First, I will present our studies on examining and mitigating societal biases in a wide spectrum of language tasks. Then, I will describe how we can identify biased predictions made by a model. I will conclude the talk by discussing my other interdisciplinary research projects and future research plans.