Go for systems operators, part 1: a brief history of Go

For Sensu Software Engineer Eric Chlebek, the Go programming language is a core part of his day-to-day, and building a monitoring tool has given him unique insight into the world of operations and what operators are looking for in a monitoring solution. In this series, he shares his learnings about Go as it relates to and benefits IT operators. This first post sets the stage with a history of Go, with the rest of the series focussing on running Go applications — including what operators can do to diagnose problems and improve performance.

A brief history of Go

By now, you may have heard about the Go programming language, or have heard about some prominent projects written in it, including etcd, Kubernetes, Terraform, and many others. Go is used by services like YouTube and Cloudflare.

Why did Go become so popular, and why is it increasingly being used to create tools for building, monitoring, and deploying information systems? To answer this question, it’s important to first understand the history of Go.

In many ways, the history of Go is tied to the history of Unix, and the computing philosophies that lead to Unix and its derivatives becoming the dominant operating system around the world today. Technologies like POSIX threads, UTF-8, and IP networking are relied on heavily in the Go standard library and in most applications written in Go.

Pre-history: Unix and the C language

In 1969, a group of Bell labs employees became frustrated with Multics, and started working on a replacement that they eventually called Unix.

In 1972, Dennis Ritchie developed C as an improvement to the B language, while working on Version 2 Unix with Ken Thompson.

In 1978, Dennis Ritchie and Brian Kernighan published the first version of “The C Programming Language”, which became the informal specification of C for several years thereafter.

In the late 1980s, Rob Pike and Ken Thompson started developing the Plan 9 OS in C, with support from others in Bell Labs, and over time, other famous computer programmers. It was released to the public in 1992. Plan 9 was never widely adopted, but still exists today as a point of interest for operating systems developers.

In 1992, Ken Thompson and Rob Pike developed UTF-8 on a placemat in a diner in New Jersey.

By the late 2000s, Ken and Rob had left Bell Labs and gone to work at Google.

In 2009, Robert Griesemer, Rob Pike, Ken Thompson, Ian Taylor, Russ Cox, Jini Kim, and Adam Langley announced that they were developing a new experimental programming language at Google, which would become Go. The Go programming language is closely related to the work they had done on Unix, C, and Plan 9. Its source encoding is UTF-8, and it is one of the first programming languages to provide first-class support for UTF-8 strings.

2012: Go 1 is released

In 2012, the first stable release of Go was made. The Go project promised that all Go programs written to the Go 1 specification would continue to work in future 1.x releases. This encouraged many developers who were evaluating Go, as they could be certain that their efforts in adopting a new programming language would not go to waste. Go included a well-rounded standard library, with good support for cryptography, networking, graphics, templating, and databases. Additionally, Go made cross-compiling relatively simple, so people could run Go programs on many different CPU architectures and operating systems.

2014: Terraform 0.1.0 is released

In 2014, HashiCorp released Terraform 0.1.0.

Before that, HashiCorp had traditionally developed their applications in Ruby, and Terraform marked a big internal toolchain shift for them. This was at a time when nearly all DevOps tools were being developed in Ruby, and Go was still considered to be an up-and-coming language.

2014: Ben Johnson releases Bolt DB version 1

In 2014, Ben Johnson released the first stable version of Bolt DB, a fast key-value embedded database, inspired by Howard Chu’s Lightning Memory Database project. At the time, the project is already used in etcd and several other Go programs.

2015: First release of Kubernetes is published

Kubernetes, a Linux container scheduler, was designed by Google. In many ways, it is similar to the Borg scheduler used internally at Google. Kubernetes is a large, complex piece of software, capable of running clusters of thousands of machines.

2015: etcd v2.0.0 is released

In 2015, the first stable version of etcd was released. etcd is a distributed key-value store that promises strict consistency, using the raft consensus algorithm. It is implemented in pure Go.

2015: Go 1.5 is released

In 2015, the first self-hosting version of the Go compiler was released. This meant that the Go compiler was compiled with Go itself. Until version 1.4, the Go compiler was compiled with C. This is often considered an important milestone for a compiled programming language, because it indicates that the language and toolchain have matured in terms of performance and features enough to be used for important tasks.

Additionally, Go 1.5 shipped with an improved garbage collector, which would pause application execution for less than 10ms when garbage was collected. This marked an important milestone for running Go in production.

A garbage collector is an automated facility that some programming languages have. It lets programmers allocate memory without worrying about releasing it back to the operating system. This tends to result in much more reliable applications, since programmers don’t need to remember to free memory, and don’t need to worry about trying to free memory twice, a classic stumbling block in C and C++.

2015: Prometheus is publicly released

Prometheus was developed at Soundcloud, and released to the public in 2015. Prometheus quickly saw widespread use, and replaced StatsD as a metrics collection tool in many organizations.

2016: Sensu adopts Go

While Sensu is a small part of the landscape when it comes to Go projects, 2016 marked a change in engineering culture at Sensu, as development on Ruby projects was de-emphasized in favour of Go.

2017: Go 1.8 is released

In 2017, Go 1.8 was released, adding even more improvements to Go’s garbage collector. As of Go 1.8, the garbage collector would pause applications for less than 1 millisecond, which had a huge positive impact on low-latency Go applications.

2018: Go 1.11 is released

The release of Go 1.11 marked the addition of Go modules. Until now, Go had no official module system, and many third parties had developed tools for solving dependency management in Go.

Go modules quickly became the standard way to handle dependencies in Go, and by Go 1.13, was the recommended approach for all projects.

2020 and beyond

Go continues to evolve, consistently refining its tooling, runtime, and language features.

Operators who pay close attention to the Go release notes will notice small, incremental improvements to the runtime and compiler. Binary size reductions, garbage collector improvements, better support for profiling, and better support for tracing. This means operators who are running Go applications get better performance, better observability, and better debugging facilities as new versions of Go are released, and applications are re-compiled.

Go continues to capture mindshare among software developers. Deploying Go applications is consistently easier than deploying Python and Ruby applications, and learning to write Go programs is very accessible to people who have a background in languages like Ruby, Python, or Javascript.

While some of Go’s intended use-cases have been under heavy competition from new languages like Rust, Go applications continue to see widespread deployment.

Conclusion

This article touched on some highlights in Go’s language development, and also covered some notable applications written in Go. Now that we’ve learned about Go and how it’s been received by the wider industry, we have set the stage for future articles for systems operators using applications written in Go. In my next post, I’ll discuss how operators can use the Go profiler to quickly inspect key aspects of running Go processes.

Keep an eye on our blog for the next post in the series, or get notified by subscribing to our newsletter!