I drink almond milk regularly. There are many brands of almond milk that don't taste great, but there are many that taste really great as well. You need to try different brands to determine which is best for you.
Dairy milk can be particularly hazardous for hormonal health due to its high levels of estrogen. Many people know that soy is high in phytoestrogen, but what is less well known is that the prefix phyto- means "plant" and this should be a strong clue that plant estrogen in soy does not work well in humans (animals). Dairy milk, on the other hand, comes from cows, which are not only animals but also mammals, just like humans. We are closely related to the cow, and the estrogen in cow's milk (estradiol) is highly potent in humans.
Men who drink dairy milk regularly suffer from a significant drop in serum testosterone levels due to high levels of mammalian estrogen (estradiol) in dairy milk.
http://www.ncbi.nlm.nih.gov/pubmed/19496976
Almond milk has zero issues with estrogen, both plant estrogen and mammalian estrogen. However, many people, when arguing against almond milk, argue that its protein content is low.
My response to this is that the protein content of food is hardly a great indicator of how healthy it is. If it were, milk would not be unhealthy compared to, say, protein shakes. In fact, if protein were the measure if how healthy food is, we should all drink protein shakes all the time since one cup of protein shake contains about 25 grams of protein compared to dairy milk, which contains only 9 grams of protein per cup.
Of course, focusing on protein is a concern if you protein deficient, but virtually no one suffers from protein deficiency in the developed world. Even if you do have a high need for protein if, say, you were a bodybuilder, milk is very inefficient because protein shakes contain more protein. Furthermore, if you were a bodybuilder, you'd be very concerned about the mammalian estrogen in dairy milk and its impact on serum testosterone levels.