Profiling Go Applications in Production

Production environments are different from development and staging. Whether optimizing application performance or trying to find a problem root cause, it is often necessary to use Go’s profiling tools in production.

Obviously, before doing anything in production, especially for performance optimizations, it makes sense to try to simulate production load and generate for example CPU or memory profiles with the built in profilers using go tool pprof.

Using built-in profilers

To profile remote applications, go tool pprof accepts URLs, e.g.:

go tool pprof http://localhost:6060/debug/pprof/profile

The endpoint should be initialized on the application, by importing the pprof http handler package:

import _ "net/http/pprof"

Also the application should have a listening HTTP server, e.g.:

http.ListenAndServe("localhost:6060", nil)

It’s important to ensure that this endpoint is not accessible for outside world, for example by only allowing access from internal addresses.

The next step would be to regularly request profiles from the application and track performance regression at the function call level or identifying root cause of a performance problem.

Using StackImpact

StackImpact takes production profiling to the next level. It automatically collects and reports regular and anomaly-triggered profiles, including:

  • CPU usage
  • Memory allocations
  • Network waits
  • System call waits
  • Locking

These profiles along with application and runtime metrics are reported to the StackImpact Dashboard, where they can be viewed and compared. Analyzing performance differences at function call level over time are critical for performance optimizations and problem root cause identification.

dashboard

It takes a few minutes to get StackImpact set up and running. Just sign up for a free account, get the agent with go get github.com/stackimpact/stackimpact-go and add these two lines to your application:

agent := stackimpact.NewAgent();
agent.Start(stackimpact.Options{
	AgentKey: "agent key here",
	AppName: "MyGoApp",
})

See documentation for detailed setup instructions.

Follow us on Twitter @stackimpact for updates on performance profiling and monitoring of production Golang apps.