Monitoring CPU Hot Spots in Production Go Applications

Identifying CPU hots spots with line-of-code precision is critical when it comes to troubleshooting performance issues, identifying performance bottlenecks or improving response for a better customer experience.

Go’s pprof toolset has powerful tools for CPU profiling and visualizing different aspects of profiles. It is very useful in the development stage. However, profiling CPU usage in production environments has different requirements:

  • We need to be able to see CPU profiling information continuously to have a historical baseline for optimization or troubleshooting.
  • Production environments increasingly become container-based and automatically orchestrated, e.g. when using Kubernetes, which implies that it might not be easy to locate and connect to a production machine to initiate the profiler remotely.
  • If the application has crashed, we need information before the fact.

Using StackImpact for automatic CPU profiling

StackImpact is designed for profiling and monitoring production environments. It completely automates the collection of CPU profiles. The StackImpact agent, which is initialized in the application, records and reports regular and anomaly-triggered CPU profiles to the Dashboard. Here is how to add the agent to the application:

Get the agent key at stackimpact.com.

go get github.com/stackimpact/stackimpact-go
agent := stackimpact.Start(stackimpact.Options{
	AgentKey: "agent key here",
	AppName: "MyGoApp",
})

See documentation for detailed setup instructions.

After restarting/deploying the application, the profiles will be available in the Dashboard in a historically comparable form.

dashboard

Similar profile history is automatically available for:

  • Memory allocations
  • Channel waits
  • Network waits
  • System call waits
  • Locking

Metrics from Go runtime are also available in the Dashboard.

Follow us on Twitter @stackimpact for updates on performance profiling and monitoring of production Golang apps.