There are as many kinds of antagonists as there are stories, but from what I’ve observed, there are two types which are particularly effective.
The first type is a sympathetic antagonist. The word ‘sympathetic’ can mean several different things, but here I mean a character that you like or even root for. Perhaps they are simply an antagonist because of an opposing goal, such as winning the same contest or enforcing the law your protagonist is breaking, but they’re not actually a bad person. Or perhaps they are just so likeable through humor or romance (a love interest can go a long way here) that the reader doesn’t want to see them defeated… all the way.
The reason a sympathetic antagonist can be effective is that the more the reader roots for them, the more torn they are between wanting your protagonist or them to win. Depending on how well you develop both the protagonist’s motivation and the antagonist’s, as well as the storytelling tools (multiple POV, foreshadowing, etc.) that lead to a reader trying to predict the outcome—you could have a scenario where the reader truly doesn’t know what will happen.
The TV show Justified uses this technique to tell both sides of the story, and does so effectively in such a way that when the hero and the ‘villain’ come up against each other, you truly don’t know who you want to win. This leads to more suspense and tension, and less predictability. The two main things to watch out for is developing the antagonist at the expense of the hero, which really just flips the story focus and leads the reader to root against the hero; and writing yourself into a corner, where you have to disappoint readers since there’s no positive outcome. In some stories, this works fine—any victory is bittersweet or dark or tragic. But if you want a happy ending, you might have to pull a rabbit out of your hat to let the hero and the antagonist win in some way (or team them up against someone even worse—a very effective technique).
The second type of effective antagonist is the heinous villain. This is the exact opposite strategy from the one above—here, the goal is to make the villain so awful that the reader cannot wait to see them stopped. The more powerful, crafty, and not-easily-stopped the villain is, the stronger the desire to see them crushed.
Heinous villains usually have more personal awfulness rather than big epic plans to destroy the world. In fact, a sympathetic antagonist could have a big epic plan, while a small-time local antagonist could be truly hateful. The reason is that characters who directly thwart, insult, aggravate, abuse, and torment your heroes are far more hated than a character who might be inflicting violence or pain on others (this isn’t necessarily right, but it’s a simple fact that the reader will care more about what happens to your hero than the rest of the world).
You can also add some other heinous flavor that’s more generic—prejudice, sexism, hypocrisy, abuse of power, etc. If we see a character behaving this way and getting away with it, we will want to see them stopped. The only danger is going too far and turning the villain into a cartoon of awfulness—depending on the tone of your story, it won’t work, and will actually make the villain less effective.
This is why many people hate Umbridge more than Voldemort in Harry Potter, despite the fact that the latter is far more harmful and violent. It’s why Joffrey Baratheon in Game of Thrones is vile, even though the adults around him are pulling the strings.
The trick for this technique is to set up your heinous villain and let them win—either because they are just too powerful, or because the heroes don’t know who they are (this is especially effective with multiple POV where we can see the villain working behind the scenes, while they play the nice guy to the heroes—see the first season of Once Upon a Time for an example). The stronger the villain seems, the more we want to see them stopped or defeated (because of how jerky they are), the more suspense your story will generate.
A great hero can make a story—but a great villain will turn pages.