It's amazing how the "harm principle" has become such a simple standard in our moral calculus. Society should strive to only limit people's rights when not doing so would result in harm to others. It's brilliant in its simplicity and makes for very easy application. However, it strikes me as something of a moral crutch in a considerably more complex universe. I think it, as a general rule, is helpful, but by its very nature delimits certain areas where intervention would certainly be justified, but doesn't fit under a black and white rubric.
The question really comes down to the definition of harm. In typical parlance "harm" has very clearly come to mean physical harm or psychological harm so damaging that its effects are apparent and the damage is devastating. Even then, this is usually limited to physical actions (like holding an unloaded gun to a person's head) rather then speech actions. It largely ignores social harms, particular those that don't effect discrete bodies of people (e.g. race and gender). The reason this seems to be the standard is that it is simple to view, we can understand very clearly how someone has been harmed when we see a severed limb, but it's much harder to see how someone is harmed internally by the type of society we live in. But nonetheless, this type of harm is quite real.
Allow me to give you something of a strawman (or strawwoman if you prefer) example: Columbine. Now, of course, this is extreme, but probably for the best in terms of illustrative quality. We have a pattern of high school kids feeling not just out of place, but totally disconnected with the community in which they lived. There are some dreadfully dire consequences when people feel left outside the cultural superstructure. The problem is that many of these harms are subjective and vary from person to person, but that doesn't mean that they are weak or avoidable. They are just as (if not more) damaging than many of the physical harms we would seek to restrict.
Now the obvious question is where does the rubber meet the road? I don't know. Social engineering is a tough chore and a somewhat creepy prospect. However, being confronted by an incredibly complicated balancing act is not necessarily the same as a reason to restrict such regulations outright. For instance, I think there is an argument to be made that restricting hate speech is totally morally acceptable; particularly if such speech has the visceral effect of denigrating individuals and making them feel displaced in our society.
I think throwing away the harm principle as a standard opens up a lot of interesting questions about how we approach regulation in our society. For instance, obscenity laws become a much richer debate. Why, in fact, do we have a presumption in our society that the rights of the creators of art overwhelm the rights of people who are made uncomfortable and less a part of our social fabric as a result of such art? I actually thing there are quite a few good arguments on both sides of this issue (tending a bit towards permitting the creation of such art), but to dismiss the claims of the offended individuals outright is both arrogant and self-destructive.
While the harm principle is a useful conceit, its underlying assumptions have lead to a lot of division in our society, a breech is almost impossible to bridge analytically (rhetorically a certain candidate has made it seem possible but, as I've made clear before, I am somewhat skeptical of its long-term viability). We favor action over harm and in turn we bias the system over those who value a certain type of life over others. We are making a choice, just because it's a choice by mechanism, doesn't mean it is not still a choice. Moreover, we allow the market free reign to regulate choices and expression. For all I want my speech rights, it begins and ends with my physical ability to express that speech. There is no natural state, thus no such thing as a natural freedom. Every time we choose to grant a liberty, it is a form of restriction being imposed by the state on others not to prevent that liberty. I obviously think rights are important, but I equally think it's important to recognize that in a socially-constructed world perception and history create those rights and that in order to have a truly fair a representative society everyone and every perspective needs to be part of the conversation. Part of that is moving off moral assumptions and the bias for action over discomfort upon which the current harm principle is predicated.