Here was my comment:
My gut feeling is that if your players aren’t scared (or at least scared for their characters) by the monsters, they don’t have enough “skin in the game.”
Step 1: Have permanent effects. If whatever the monsters do (perhaps even including killing the characters) can be reversed, it’s just not as scary. The easier it is to fix, the less scary, but as long as there’s a known fix, getting effected goes from horror to nuisance. The monsters are just bags of hit-points when the PCs are just bags of hit-points that can be restored with a couple spells or some rest. Monsters that you want to be scary should have some good chance of permanently changing the characters for the worse. Death counts if there’s no resurrection, but too much and the players will just avoid identifying with the characters if they still agree to play at all. Level Drain, destroying stats (e.g. the slashing, filthy claws of a Harpy disfigure the victim and remove 1 point of CHA on a critical hit), inflicting curses, chronic disease, phobias, destruction of precious possessions (yay, rust monsters!) are all ways to make the players fear the outcome of the battle even if “victorious.”
Step 2: Be Unfair. Fairness isn’t scary. It may or may not be exciting, but if you think you’re evenly matched, you may feel bad about losing, but you won’t dread even getting in the fight. You gave it the old college try. Monsters are scary when you’re their prey, not their equal. If your harpies are too tough for anyone under 20th level to take on (perhaps they really are death spirits embodied, as in Greek mythology, and immune to missile fire and spells), or just too numerous when they do show up, then you can get your players to cower under the trees whenever a flight of them wheels through the sky.
Step 3: Use sparingly. Unless you’re running Call of Cthulhu you probably want your players to play brave adventurers. It’s easy to accidentally overdo it and cause them to “turtle” because the outside world is just too scary.
I’ve been thinking about fear in RPGS a bit recently, but I’ll save that for another post.