At some point in your writing, you will end up writing some form of magic. There are a few tricks you can do to make the magic interesting and fascinating to read.
First, leave off the idea that there is any set type of magic you'll need. If you're writing Willow and Tara doing Wiccan magic, you are going to have to take the time and do a little research if you're going to make it believable. But aside from that, feel free to play. After all, even Joss doesn't hold to a specific convention when it comes to magic, and neither should you.
Now that you're ready to experiment, start thinking about these questions:
What do you need the magic to accomplish?
How much of your story is focused on the magic? Is it simply a small plot device or is it a major facet of your story?
What sort of tone do you want to be set by the magic? Is the spell supposed to feel dangerous, dark, sexy, joyous, powerful, overwhelming, etc.?
Once you have a clear idea of what your magic is going to be, you can start laying the foundation down for it. If it's simply a device to move the plot along, then the description and explanation doesn't have to be any longer than a few lines -- but make sure those few lines are clear. Your best guide for this is the show itself, listening to what Joss and company considers essential information.
If the magic is needed to accomplish something major, and it's a large portion of the plot, then it's going to require some serious planning, explaining and build up in your story. You need to plan it out before you start writing. A few things that you should consider as you start to determine the structure of the spells and rituals are:
Where is the magic coming from? What is its source?
Who provides the information and knowledge of the spell? Why does that character have that information?
What is the source of the spell's power?
Who is involved with the spell?
Who is the caster? Why is this person the caster?
Who is affected? Why are they affected?
What are the consequences? Is the spell long-term?
Are there after-effects? Will those involved retain those effects for a long time or are they short-lived?
Will it drag out other emotions?
Most of this will be determined by the plot of your story. But it could have serious consequences on how the characters work with the magic. If the magic is some sort of arcane knowledge that one of the characters knows, then the magic will be carried out with less confidence by the other characters than a ritual that's been performed before. If a character has done the magic before, then what sort of associations does the ritual have?
Also, the source of the magic will affect things. If it's a part of Willow's witch powers or Buffy's slayer heritage, then it would probably have a greater feeling of urgency or personal connection than a ritual which could be done by any person. It usually makes sense for the person with the knowledge of the spell to perform the spell, but often this isn't what happens. Try to have a compelling reason for the caster to be performing the spell. The stronger the reason, the more interesting the magic.
Making the magic feel like magic can be tricky. Think about a few things before you begin:
What is the magic's structure?
What supplies are involved?
What's the format of the ritual?
How does it progress?
There are usually a few things which are common to most written magic. An incantation of some sort, physical ingredients, usually some flash and bang, and the caster experiencing some force. Absolutely none of these are essential to writing magic. But you will probably want to use some of these things in some combination, so think about what you're using and why you're using it.
Much of the structure of the ritual should be determined by the things you've already considered, but make sure the structure of the ritual is reflective of what your magic is trying to accomplish. If, for example, your magic is to give Buffy greater power, then part of the ritual should be for her to undergo some test to prove her courage or willingness to sacrifice. If a bond is being formed, then both people should be involved in the ritual.
The incantation holds a special importance -- it explains the influences, reasons, and actions involved in the ritual. Make sure your incantation does three things:
Offers up the objective. ("I ask that you heal this being...")
Involves/mentions/addresses the caster and the person affected. ("My name is Rupert Giles, son of...and I ask that you take Willow Rosenberg, daughter of...")
Is reflective of the tone you are trying to set. ("Blood and ashes surround you, and I ask for the death of...")
While you're thinking about the tone you want to set, include the actions and special effects of the ritual. If you want the magic to feel dark or sexy, then you could minimize the use of light and mention the shadows...
Your choice of ingredients, such as herbs, candles, and focus objects, can go a long way to setting the mood as well. Common focus objects are jewelry, weapons, other people, and magic symbols. Watch out for just brushing them off as some arbitrary item. The higher the stakes and attachments involved create a more interesting and intense piece. If you've decided that your ritual has some great personal meaning to the people involved, then keep this personal meaning with your choice of ingredients. Instead of the spell requiring just a doll, it could require Buffy's favorite stuffed animal. Or a spell involving Angel could involve a cross, which has elements of destruction and redemption for him.
The climax of the magic won't be determined by the special effects or what's said in the incantation -- it comes to what the caster experiences at the height of the ritual's power. Think of the spell done by Willow and Tara during "Who Are You." What made that spell so compelling wasn't what was said, or done, but the effect that it had on Willow. If you focus on what the caster feels, the same sort of compelling force will flow through your story as well.
The last thing you'll want to think about is how complicated you want the ritual to be, and why it's that complicated. If it involves only a few ingredients and incantations, is it because it's a relatively simple spell, or is it because the caster is so powerful or so used to this spell that it requires little guidance? If it involves a lot of work, is it because it's a big spell, or that the people involved have to learn everything from scratch?
Now that you've considered all the aspects of your magic, it's time to include it in the story.
The first step is always build-up. You've already created the reasons for performing it, but you also have to make sure your readers have enough information to understand what you're doing in the ritual. This information will come from the story, but you have to be careful on how you do it. Long-winded explanations can bore even the most patient people, and you have to keep people actively reading the story. The more emotion involved in the explanation, then the more interesting the explanation will be. Some possible methods are:
The person with the knowledge of the spell has the knowledge because of their past, and the explanation brings forth memories.
The explanation is short and concise because of the urgency of the spell.
The explanation takes place at the same time that the spell is being performed.
Another character reacts as the spell is being explained.
Don't explain the spell until after it's performed.
The actual writing of the spell is a chance to work with incredibly rich and unique imagery. Keep in mind the tone you're trying to set with the spell when you choose your words. Also decide which portions of the spell you want the reader to pay special attention to.
Depending on where you want the emphasis to be placed, that's where you'll put the most description. If you are using ingredients or focus objects, then take the time to give the readers a clear picture of those objects in relationship to the characters. Think in terms of things other than sight to enrich the description. If you're using herbs, don't just give the names of the herbs, talk about the texture or smell. Make the readers feel like they're there.
When writing incantations, it would be helpful to write them in English, but including another language may add some depth and mystery to the spell. You may want to include one of the characters mentally translating the spell, however, so that anything important in the incantation isn't lost to the readers. Also, write the incantations like dialogue -- don't ignore inflection and actions when writing them, or else they come out like a long, dry speech.
Also, try to keep things as simple as possible. You may know what all the terms and concepts are, but some of them might be confusing to others, and if you're not as well-versed as others, you may be getting part of the incantation wrong. Focus on what the characters are feeling, both physically and emotionally, and the spell should read as perfect as it can be.