15 — Concurrency Patterns — Pipelines, Fan-Out, and Fan-In

Composing goroutines and channels reveals recurring shapes, and naming them is what turned concurrency from guesswork into design. The model that clicked: a pipeline chains stages where each stage is a goroutine reading from one channel and writing to another; fan-out distributes one input across many workers; fan-in merges many inputs into one channel — and these three compose into most real concurrent designs. [1] Running underneath is Go's error model — errors as values, wrapping with %w, sentinels with errors.Is, and panic/recover as the rarely-used safety net for the genuinely unrecoverable [4]. The patterns plus the error model together are how Go concurrent programs are written.

The pipeline — stages connected by channels

A pipeline is a series of stages, each a goroutine that reads from an inbound channel, transforms the data, and writes to an outbound channel [1]. The classic shape:

// Stage 1: generate
func gen(nums ...int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for _, n := range nums { out <- n }
    }()
    return out
}

// Stage 2: square
func square(in <-chan int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for n := range in { out <- n * n }
    }()
    return out
}

// Composed
ch := square(square(gen(2, 3)))   // 2→4→16, 3→9→81

The power is concurrency: each stage runs in its own goroutine, so while stage 1 is producing the next value, stage 2 is processing the previous one. The channels between them synchronize automatically. Closing the outbound channel with defer close(out) when a stage finishes is what lets the downstream range terminate — the close propagates down the pipeline [1]. The pattern scales from a two-stage transform to a multi-service data pipeline where each stage is a different concern.

Fan-out — one source, many workers

Fan-out takes a single input channel and distributes its items across multiple worker goroutines, each processing independently [2][3]:

inputs := gen(1, 2, 3, 4, 5, 6)
c1 := worker(inputs)
c2 := worker(inputs)
c3 := worker(inputs)
// three goroutines now consume from the same channel in parallel

Because a channel is safe for concurrent receive, multiple goroutines pulling from one input channel automatically share the load. Fan-out is the answer to "this CPU-bound stage is my bottleneck" — spin up N workers, let them race for items off the input channel, and throughput scales with N (up to the number of CPU cores for CPU-bound work). It is also the building block of the worker pool pattern from the goroutines notes.

Fan-in — many sources, one output

Fan-in is the mirror: it merges multiple input channels into a single output channel [2][3]:

func merge(cs ...<-chan int) <-chan int {
    out := make(chan int)
    var wg sync.WaitGroup
    wg.Add(len(cs))
    for _, c := range cs {
        go func(c <-chan int) {
            defer wg.Done()
            for v := range c { out <- v }
        }(c)
    }
    go func() { wg.Wait(); close(out) }()
    return out
}

A sync.WaitGroup (covered in the next notes) tracks the input goroutines; once all have finished, a separate goroutine closes the output so the downstream range terminates. Fan-in is how you re-merge the results of a fan-out — split work across N workers, then merge their N output channels back into one. The combination fan-out → workers → fan-in is the standard structure for parallel processing with ordered completion handling.

The error model underneath

Concurrency patterns move values; the error model decides how failures move with them. Go's philosophy: errors are ordinary values, returned explicitly, never thrown. [4]

result, err := doWork()
if err != nil {
    return fmt.Errorf("doWork failed: %w", err)   // wrap, preserving the chain
}

Four pieces compose the model:

• errors.New("msg") creates a simple static error [6].
• fmt.Errorf with %w wraps an existing error, adding context while preserving the original for inspection [7].
• Sentinel errors (io.EOF, sql.ErrNoRows) are predefined package-level values, checked with errors.Is(err, io.EOF) [8].
• errors.As extracts a typed error from a chain, useful for custom error types that carry data.

The discipline is: check err != nil at every layer, wrap with context as you bubble up, and let the top level decide what to log versus surface to the user. No exceptions means no hidden control flow — every failure path is visible in the function signatures.

panic and recover — the safety net, not the strategy

Go does have a panic mechanism, but the language culture reserves it for the genuinely unrecoverable: an invariant violation, a nil pointer the programmer should have prevented, an index out of range [9]. A panic unwinds the stack, running deferred functions; a recover inside a deferred function can catch it and turn it into a normal error:

defer func() {
    if r := recover(); r != nil {
        log.Printf("recovered from panic: %v", r)
    }
}()

The rule from every style guide: do not use panic for ordinary error handling. If a function can fail in an expected way (file not found, network down, bad input), it returns an error. Panics are for bugs and impossible states. The main legitimate use of recover is at a goroutine boundary — a panicking goroutine kills the whole program, so a top-level recover in long-running server code prevents one bad request from taking down the process. Combined with stack traces (printed automatically on panic) and the delve debugger, this gives the diagnostics needed to fix the root cause.

How I use this

The pipeline/fan-out/fan-in vocabulary is how I structure any concurrent processing — a read stage feeds a worker stage feeding a write stage, with fan-out at the bottleneck and fan-in to collect. Cancellation flows through every stage via context, so the whole pipeline shuts down cleanly on timeout or shutdown signal. On the error side, every function that can fail returns an error, I wrap with %w at each layer to build a causal chain, and I reserve recover for exactly one place per long-running goroutine — the top of a request handler or a worker loop — so a single panic cannot kill the process. The result is concurrent code that fails loudly and locally, and whose error messages explain the full path from symptom to cause.

References

[1] S. Cox, "Go Concurrency Patterns: Pipelines and cancellation," The Go Blog, 2014. [Online]. Available: https://go.dev/blog/pipelines

[2] GoLinuxCloud, "Fan Out Fan In Concurrency Pattern Explained," 2024. [Online]. Available: https://www.golinuxcloud.com/go-fan-out-fan-in/

[3] Geek Culture, "Golang Concurrency Patterns: Fan in, Fan out," Medium, 2021. [Online]. Available: https://medium.com/geekculture/golang-concurrency-patterns-fan-in-fan-out-1ee43c6830c4

[4] The Go Authors, "Error handling and Go," The Go Blog, 2011. [Online]. Available: https://go.dev/blog/error-handling-and-go

[5] Gopinath R., "Go Concurrency Patterns: A Deep Dive," Medium, 2024. [Online]. Available: https://medium.com/@gopinathr143/go-concurrency-patterns-a-deep-dive-a2750f98a102

[6] DigitalOcean, "Creating Custom Errors in Go," 2024. [Online]. Available: https://www.digitalocean.com/community/tutorials/creating-custom-errors-in-go

[7] V. Kareem, "Golang: error wrapping / unwrapping," Medium, 2024. [Online]. Available: https://medium.com/@vajahatkareem/golang-error-wrapping-multierror-759d04bdbfaf

[8] Tired SG, "Golang Sentinel Error," 2024. [Online]. Available: https://www.tiredsg.dev/blog/golang-sentinel-error/

[9] The Go Authors, "Defer, Panic, and Recover," The Go Blog, 2010. [Online]. Available: https://go.dev/blog/defer-panic-and-recover