Profiling HTTP Handler Latency in Go

Profilers are essential tools for understanding which code causes high CPU, I/O or memory consumption. Generated profiles represent an aggregated view of application execution, regardless of the type of work the server is doing. In latency-critical applications, the latency usually has a higher importance than resource consumption in terms of user experience or other business metrics.

Using full-program profiles to optimize server latency is challenging and often impossible. However, when a profile only represents one type of work or an event (e.g. single HTTP handler), it can be instrumental in server latency optimization.

Recent pprof’s support for labels allows CPU profiler to group the samples it records by given labels. This in turn makes it possible to generate handler-specific profiles. Now, using labels, each HTTP handler can have its own profile. We can use these handler-only profiles to analyze the response times of specific handlers.

StackImpact Go agent uses labels under the hood to generate HTTP handler profiles in addition to the full-process profile. Here is an example that uses the agent’s handler wrappers to generate per-handler workload profiles:

package main

import (
	"fmt"
	"math/rand"
	"net/http"

	"github.com/stackimpact/stackimpact-go"
)

func requestHandler(w http.ResponseWriter, r *http.Request) {
	// simulate cpu work
	for i := 0; i < 10000000; i++ {
		rand.Intn(1000)
	}

	fmt.Fprintf(w, "Done")
}

func main() {
	// Initialize StackImpact agent
	agent := stackimpact.Start(stackimpact.Options{
		AgentKey: "your agent key here",
		AppName:  "Workload Profiling Example",
	})

	// Serve
	http.HandleFunc(agent.ProfileHandlerFunc("/test", requestHandler))
	http.ListenAndServe(":9000", nil)
}

This is what the profiles for the /test handler looks like in the StackImpact Dashboard, in the Bottlenecks/Latency section after generating some requests to our example application:

Screenshot

Try it out yourself! It takes a few minutes to setup the agent. See instructions on the agent’s GitHub page.