7 quiet ways toxic people drain your energy without saying a word

You know those people who leave you feeling oddly tired, even though they barely said a thing?

It’s not your imagination. Toxic people don’t always need to speak to pull the life out of you. Sometimes, their silence says more than words ever could.

I’ve experienced this in friendships, work settings, and even during yoga retreats where the vibe is supposed to be calm and nurturing. What looks like nothing on the surface can be deeply draining if you don’t recognize it.

And once you do, you realize something important: silence itself isn’t the problem—it’s the intent behind it. Silence can be nourishing, like sitting quietly with a trusted friend. Or it can be heavy, manipulative, and exhausting.

Here are seven subtle ways toxic people use silence and nonverbal cues to quietly chip away at your energy.

1. Silent judgment

Have you ever sat next to someone and felt them sizing you up without a word? Maybe it was a slight smirk when you spoke, or an eyebrow raised when you wore something different.

I’ll never forget a dinner party where I felt this so strongly. A woman across the table didn’t speak much, but every time I shared a story, she gave a tight smile and quick downward glance. She didn’t need to say, “That’s silly.” Her face said it for her.

That kind of silent judgment makes you second-guess yourself. You start editing your stories, toning down your enthusiasm, or checking if you said something “wrong.”

As noted by psychologist Dr. Susan Krauss Whitbourne, “Nonverbal judgment can sting more than words because it’s harder to confront.” When someone openly criticizes you, you can respond or set a boundary. When they simply radiate disapproval, you’re left stewing in self-doubt.

If you’ve felt this, ask yourself: is this my insecurity, or am I picking up on someone else’s energy? Sometimes it’s both—but either way, you don’t need to shrink to make someone else comfortable.

2. Withholding warmth

I once had a colleague who never greeted me with a smile, even when everyone else did. She wasn’t overtly rude—just consistently cold, detached, and unapproachable.

That lack of warmth spoke louder than words. I found myself wondering if I had done something wrong. And that’s exactly the trap. When someone withholds basic kindness, it creates a power imbalance. You end up chasing their approval, hoping they’ll finally thaw.

Energy flows where connection grows. When someone refuses to meet you halfway, you’ll notice your own energy being quietly drained.

This is why I now pay close attention to how I feel around people. Do I leave an interaction lighter, or heavier? As mindfulness teacher Thich Nhat Hanh said, “The most precious gift we can offer anyone is our attention.” When someone intentionally withholds it, that’s a red flag.

3. Using the silent treatment

Silence can be weaponized.

Toxic people sometimes withhold conversation or affection to punish or control. They don’t slam doors or shout—they just go quiet and watch you squirm.

I once experienced this in a relationship years ago. Anytime I brought up something he didn’t want to discuss, he’d go completely silent. Sometimes for hours, sometimes for days. I’d feel anxious, scrambling to fix whatever I had done. Looking back, I see it for what it was: emotional manipulation.

Communication coach Preston Ni explains it clearly: “The silent treatment is a form of emotional manipulation designed to make the other person feel guilty or anxious.” It’s not about peace or space—it’s about control.

Healthy silence is about respect. Toxic silence is about punishment. Learn to tell the difference.

4. Constant sighs and eye-rolls

Words aren’t necessary when someone wants to make you feel small. Heavy sighs, exaggerated eye-rolls, or dramatic pauses can cut deeper than criticism.

I once shared an idea in a team meeting, only to be met with a loud sigh from someone across the table. He didn’t say a word—but the message was clear: That’s stupid. For the rest of the meeting, I stayed quiet, holding back ideas I was excited about.

These nonverbal cues communicate disapproval or impatience, without the honesty of direct words. That’s what makes them so toxic: you’re left interpreting instead of engaging.

If you notice this happening often, ask yourself: do I really want to keep shrinking to make space for someone else’s sighs? Probably not.

5. Refusing to celebrate you

I remember when I got certified as a yoga instructor. Most people in my life were excited for me—except one friend who stayed silent. No “congratulations,” no acknowledgment. Just nothing.

That quiet dismissal can hurt more than open criticism. When toxic people withhold recognition, it’s often because your success threatens their fragile sense of self.

Psychologist Dr. Judith Orloff has noted that emotionally draining people often “deny you validation as a way to keep the focus on themselves.” If they can’t be the center, they’d rather say nothing at all.

Research supports this: a study found that people who perceive emotional invalidation—including withholding recognition or dismissing others’ accomplishments—report lower emotional self-efficacy.

A lack of support isn’t neutral. It’s a statement. And while it can sting, it also teaches you something: your worth isn’t dependent on who claps for you.

6. Invading your space

Sometimes silence is paired with presence that feels off. Standing too close, staring too long, or hovering without reason can leave you unsettled.

I once had an acquaintance who lingered just a little too close whenever we talked. He didn’t say anything inappropriate, but I always left those conversations feeling uneasy. My body was telling me something my brain couldn’t quite articulate: my space was being violated.

Body language expert Joe Navarro notes that “uninvited proximity is often a form of dominance or control.” When someone uses silence and presence to unsettle you, it chips away at your sense of safety and ease.

Your nervous system knows the truth before your mind does. Listen to it.

7. Absorbing without giving back

Finally, there are those people who sit with you, listen, nod—and never offer anything in return. They soak up your energy, your stories, your kindness, yet remain emotionally absent.

It’s easy to excuse this as introversion or shyness, but over time, you notice the imbalance. You walk away feeling lighter after talking to some people—and completely drained after talking to others.

I used to meet a friend for coffee who was like this. I’d leave the café exhausted, like I had just given a two-hour therapy session. She rarely asked about me, and when she did, her responses were flat. It wasn’t silence—it was emotional withdrawal.

Author and therapist Nedra Glover Tawwab has said, “One-sided relationships are sustained only by one person’s energy.” When you’re the one doing all the giving, silence becomes heavy. That’s your cue it’s time to step back.

Final thoughts

Not every quiet person is toxic. Some are thoughtful, introverted, or simply comfortable with silence. The difference is how you feel after being around them.

If you walk away calm, refreshed, or seen, that’s healthy silence. If you walk away tense, insecure, or empty, that’s draining silence.

Protecting your energy doesn’t always mean confrontation. Often, it means noticing the subtle signals, trusting your gut, and choosing distance when needed.

One mindfulness practice I use after being around draining people is a simple grounding breath: inhale deeply, exhale slowly, and picture yourself letting their energy go. It sounds small, but it clears the residue of interactions that would otherwise stick.

You deserve to spend time with people whose presence fills, not depletes, your spirit. Pay attention to what your body tells you after silence. That’s where the truth lives.