Concurrency Bugs: Mutexes, Atomics, and Lock-Free Pitfalls
Concurrency bugs are, hands down, the hardest defects to find, reproduce, and fix. They hide behind timing windows, disappear the moment you attach a debugger, and come roaring back under production load. In this post, I want to go beyond basic race conditions and dig into the subtle pitfalls of mutexes, atomic operations, and lock-free data structures, the tools developers reach for when they think they're writing safe concurrent code. The more I researched these patterns, the more I realised these tools create a false sense of security that can be worse than having no synchronization at all.
Mutex Pitfalls
Mutexes are the most common synchronization primitive, but they get misused in ways that create bugs far worse than the race conditions they were meant to prevent.
Go, Forgetting to Unlock
package main
import (
"fmt"
"sync"
)
type SafeCounter struct {
mu sync.Mutex
v map[string]int
}
func (c *SafeCounter) Inc(key string) {
c.mu.Lock()
// VULNERABLE: if the map is nil, this panics while holding the lock
c.v[key]++
c.mu.Unlock()
}
func (c *SafeCounter) Value(key string) int {
c.mu.Lock()
defer c.mu.Unlock()
return c.v[key]
}
This pattern trips up even experienced Go developers. If c.v is nil, the Inc method panics after acquiring the lock but before releasing it. Every subsequent call to Inc or Value deadlocks forever. Notice how Value uses defer correctly, but Inc does not, that inconsistency is a red flag worth looking for in reviews.
func (c *SafeCounter) Inc(key string) {
c.mu.Lock()
defer c.mu.Unlock() // SAFE: always unlocks, even on panic
if c.v == nil {
c.v = make(map[string]int)
}
c.v[key]++
}
Java, Lock Ordering Deadlocks
public class TransferService {
public void transfer(Account from, Account to, double amount) {
synchronized (from) {
synchronized (to) {
from.debit(amount);
to.credit(amount);
}
}
}
}
This is a textbook deadlock. If thread A calls transfer(accountX, accountY, 100) while thread B calls transfer(accountY, accountX, 50), thread A locks accountX and waits for accountY, while thread B locks accountY and waits for accountX. The worst part is that it works fine in testing and only shows up under production load with real concurrent transfers. I ran into a variant of this in a code review, the fix is always the same: establish a consistent lock ordering.
public class TransferService {
public void transfer(Account from, Account to, double amount) {
// SAFE: always lock in consistent order by account ID
Account first = from.getId() < to.getId() ? from : to;
Account second = from.getId() < to.getId() ? to : from;
synchronized (first) {
synchronized (second) {
from.debit(amount);
to.credit(amount);
}
}
}
}
Atomic Operation Pitfalls
Atomics guarantee individual operations are indivisible, but developers frequently assume atomics provide more guarantees than they actually do.
C++, Non-Atomic Compound Operations
#include <atomic>
#include <thread>
#include <vector>
std::atomic<int> counter{0};
void increment_if_below(int limit) {
// VULNERABLE: check-then-act is not atomic
if (counter.load() < limit) {
counter.fetch_add(1);
}
}
int main() {
std::vector<std::thread> threads;
for (int i = 0; i < 100; i++) {
threads.emplace_back(increment_if_below, 50);
}
for (auto& t : threads) t.join();
// counter can exceed 50
printf("counter = %d\n", counter.load());
}
Each individual atomic operation (load, fetch_add) is atomic, but the combination is not. Between the load and the fetch_add, another thread can increment the counter past the limit. This pattern shows up in rate limiters and connection pools, and it's broken every time. The counter can blow right past the limit.
void increment_if_below(int limit) {
int current = counter.load();
while (current < limit) {
// SAFE: compare_exchange retries if another thread modified counter
if (counter.compare_exchange_weak(current, current + 1)) {
break;
}
// current is updated to the actual value on failure
}
}
Rust, Ordering Mistakes
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::thread;
static DATA: AtomicUsize = AtomicUsize::new(0);
static READY: AtomicBool = AtomicBool::new(false);
fn producer() {
DATA.store(42, Ordering::Relaxed);
READY.store(true, Ordering::Relaxed); // VULNERABLE: Relaxed ordering
}
fn consumer() {
while !READY.load(Ordering::Relaxed) {}
// On some architectures, DATA may still read 0 here
let value = DATA.load(Ordering::Relaxed);
assert_eq!(value, 42); // Can fail on ARM/POWER
}
This one is sneaky. Relaxed ordering provides no guarantees about the order in which stores become visible to other threads. On x86 this often works by accident due to the strong memory model, but on ARM or POWER architectures, the consumer can see READY == true while DATA is still 0. What makes this particularly treacherous is that it passes all tests on x86 dev machines and only fails on ARM production servers, a pattern that shows up in bug reports more often than you'd expect.
fn producer() {
DATA.store(42, Ordering::Relaxed);
READY.store(true, Ordering::Release); // SAFE: Release ensures prior stores are visible
}
fn consumer() {
while !READY.load(Ordering::Acquire) {} // Acquire pairs with Release
let value = DATA.load(Ordering::Relaxed);
assert_eq!(value, 42); // Guaranteed correct
}
Lock-Free Data Structure Pitfalls
C++, The ABA Problem
#include <atomic>
template<typename T>
struct LockFreeStack {
struct Node {
T data;
Node* next;
};
std::atomic<Node*> head{nullptr};
void push(T value) {
Node* new_node = new Node{value, nullptr};
new_node->next = head.load();
while (!head.compare_exchange_weak(new_node->next, new_node)) {}
}
T pop() {
Node* old_head = head.load();
// VULNERABLE: ABA problem
while (old_head &&
!head.compare_exchange_weak(old_head, old_head->next)) {}
T value = old_head->data;
delete old_head;
return value;
}
};
The ABA problem sounds theoretical until it bites you. Thread A reads head = X, gets preempted. Thread B pops X, pops Y, pushes X back (possibly with different data). Thread A resumes, sees head is still X, and the compare_exchange succeeds, but X->next now points to freed memory. Solutions include tagged pointers (adding a version counter), hazard pointers, or epoch-based reclamation, and getting any of them right is extraordinarily difficult. This is why the strong recommendation is to avoid writing your own lock-free data structures.
Go, Subtle Channel Misuse
func fanIn(channels ...<-chan int) <-chan int {
out := make(chan int)
var wg sync.WaitGroup
for _, ch := range channels {
wg.Add(1)
// VULNERABLE: ch is captured by reference, not by value
go func() {
defer wg.Done()
for v := range ch {
out <- v
}
}()
}
go func() {
wg.Wait()
close(out)
}()
return out
}
This example looks perfectly reasonable at first glance. All goroutines capture the same ch variable. By the time they execute, ch points to the last channel in the slice. All goroutines read from the same channel while other channels are never drained. It's one of Go's most common concurrency gotchas, and it shows up in code reviews regularly.
func fanIn(channels ...<-chan int) <-chan int {
out := make(chan int)
var wg sync.WaitGroup
for _, ch := range channels {
wg.Add(1)
go func(c <-chan int) { // SAFE: ch passed as parameter
defer wg.Done()
for v := range c {
out <- v
}
}(ch)
}
go func() {
wg.Wait()
close(out)
}()
return out
}
Detection Strategies
Static Analysis
- Go:
go vetdetects some loop variable capture bugs. The race detector (go test -race) finds data races at runtime, it should be part of every Go project's CI.staticcheckflags mutex misuse. - C++: ThreadSanitizer (TSan) detects data races at runtime. Clang's
-Wthread-safetyannotations enable compile-time lock analysis. - Rust: The borrow checker prevents most data races at compile time, which is one of Rust's strongest selling points for concurrent code.
cargo clippywarns about suspicious atomic ordering. - Java: SpotBugs detects some synchronization issues. FindBugs flags inconsistent locking.
Manual Review
A checklist for reviewing concurrent code:
- For every mutex lock, verify there is a corresponding unlock on every code path, including error and panic paths. Prefer
defer(Go),try-finally(Java/Python), or RAII guards (C++/Rust). - For every pair of atomic operations that must be logically atomic together, verify they use compare-and-swap or are protected by a mutex.
- For lock-free code, look for the ABA problem: any compare-and-swap on a pointer that could be freed and reallocated.
- Check memory ordering on atomics.
Relaxedis almost never correct for inter-thread communication. - Verify lock ordering is consistent across all code paths to prevent deadlocks.
Remediation
The principles that the research and real-world bug reports consistently reinforce:
- Use
defer/RAII for lock management. Never manually pair lock/unlock calls across complex control flow. - Prefer higher-level abstractions. Channels (Go),
tokio::sync(Rust),java.util.concurrent(Java) are safer than raw mutexes and atomics. - Use
compare_exchangefor compound atomic operations. A load followed by a store is never atomic. - Default to
SeqCstordering unless you have a proven performance need for weaker ordering and understand the memory model implications. The performance difference rarely matters, and the correctness difference always does. - Test with race detectors enabled. Go's
-race, C/C++ ThreadSanitizer, and Java's-XX:+UseThreadSanitizercatch bugs that code review misses. - Avoid writing lock-free data structures. Use battle-tested libraries. Lock-free code is extraordinarily difficult to get right, and the consequences of getting it wrong are subtle and severe.