The power of proposals (and open source culture)

I come from a world where strategy is best kept secret. Whether it be from a company who has a codename for literally everything, or the competitive culture of playing and coaching D1 athletics, confidentiality became a required skill. Meetings, trainings, code reviews, scouting reports… anything of significance happened behind closed doors.

In other words, definitely not open source.

So after joining Sensu in late 2017, I was honestly blown away by the dependencies on and relationships with other open source companies, the healthy discussions happening between engineers and community members in Slack and GitHub, and the innovative processes and methodologies used to drive decisions. So much so, that when it came down to porting a feature to Sensu Core 2.0, I didn’t even realize that the best way to solve the problem could actually be to change the requirements.

Adapting to feedback

Check hooks are a feature in Sensu Core 1.x that are defined within the check configuration scope. Now, Sensu 2.0, which is written in Go, could have definitely supported this behavior within the check’s protobuf. However, after spending some time understanding how we defined name uniqueness in hooks, I became increasingly more convinced that it would be easier and safer to implement an in-memory etcd store. But my inkling of curiosity couldn’t possibly justify a breaking change, right?

It wasn’t until I bothered my fellow engineer Simon Plourde for help that I realized what other tremendous upside this change could bring. He talked about resource reusability and how much easier it would be for those who don’t use configuration management. He also had examples of how other resources in 2.x behaved similarly. We finally put our proposal together, and Sean Porter, our CTO and author of Sensu 1.x, had even more to add! Sean wanted to use this change to enhance auto-remediation tasks, and allow hooks to respond to mutators and handlers, too. I had no idea that by simply questioning why we were doing something the way we were (on a platform open to the public, nonetheless) could possibly have this snowball effect!

Monitor (and auto-remediate) all the things

Here’s a brief overview on the new and improved hooks that were a product of the proposal process, and shipped with our beta release (oh, did I mention we released Sensu 2.0 Beta earlier this month?!).

A 2.x check has the field check_hooks that contains a list of structs which map all the hooks for a given type to their corresponding response level:

{
    "name": "nginx_process",

   "command": "check_process.rb -p nginx",

   "interval": 30,

   "subscriptions": [ "sys" ],

   "check_hooks": [
        { "critical": [ "restart_nginx", "system_snapshot" ] },

        { "non-zero": [ "ps_aux" ] }
    ]

}

A 2.x hook is its own independent resource which is stored outside of the check configuration scope, and can be used on multiple different checks:

{

   "name": "restart_nginx",

    "command": "sudo systemctl start nginx",

    "timeout": 60

}

And don’t worry, 2.x hooks were designed with reusability in mind! Those same hooks will also be able to respond to both mutators and handlers. These features haven’t been implemented yet, but be sure to keep and eye on the following issues and future releases for that functionality.

Mutator Hooks · Issue #701 · sensu/sensu-go

Extend hook functionality to mutators! A mutator hook will respond to mutator results in the same way that a check hook…

Handler Hooks · Issue #702 · sensu/sensu-go

Extend hook functionality to handlers! A handler hook will respond to handler results in the same way that a check hook…

Open (source) collaboration is my new favorite thing

The entire feature development process for the Sensu 2.0 Beta has been elaborately documented, discussed, and implemented all on an open platform. I honestly can no longer imagine a successful scenario arriving to that point any other way. This ideology of secrecy, that was originally planted into my brain through years of sheer and brute force, now seems unnecessary and restrictive.

We’re constantly trained to trust the process, buy in all the way, and serve blindly. But in reality, this kind of open communication is what incites innovation. Engineers leveraging each other, challenging each other, and collaborating with each other achieve something far greater than they could have on their own. It’s things like Golang’s proposal process (which heavily inspires Sensu’s proposal process) and other guidelines in the open source community that help shape the culture of open source companies, enabling them to be mutually beneficial, contributing to a thriving and balanced ecosystem.

I think there are some lessons to be learned from the concept of “open source.” I’m not saying that private companies should post billboards of their proprietary information. And I’m definitely not saying that coaches should email their scouting reports and game plans to all the opposing teams in their conference. But, I think there is merit to adopting some open source behaviors and methodologies to motivate heavy communication within a team, encourage others to question the process, and to learn from and collaborate with those outside your team. Not a bad start to total transparency.

Thanks to Matthew (Brender) Broberg, Anna MacLachlan, and Nikki Attea.