GoChef!

i.e. Chef in Go -- primarily Goiardi

This session was also supposed to be about "alternate Chef Server implementations" like Opsworks but nobody wound up talking about that.

Location

Thursday, Leschi, 1430

Convener

• AJ Christensen (fujin)
• Jeremy Bingham (captain_tenille)

Participants

Among others,

• Miah Johnson
• Ranjib Dey
• Jason Cook
• Julian Dunn (Chef) - scribe
• John Ewart (Chef)

Summary of Discussions

• fujin was writing a Go Chef server before Jeremy was, but then Jeremy was further ahead

• use cases?

  • server side - hard to extend Erlang server (Ranjib)
  • in-memory, one line startup (goiardi ...)
  • easier to prototype server-side endpoints example Policyfile reference implementation
  • example: ability to implement /universe endpoint before Opscode was able to

• Jason - client library use cases are interesting.

  • Ruby is too big, good for embedded situations like Raspberry Pi, beaglebone, routers...

• Jason Cook - server-based constraint/permission system, where nodes can ask whether the server whether they're able to run

• fujin: this sort of hearkens back to Ben Black's experiment from a while ago, opscode-agent

  • state daemon running everywhere that would send state as it was requested & respond to state changes as quickly as possible
  • 4GL reactive state management - output of a state & then reaction, instead of slow Ruby code that runs forever to do parsing and then does some shit
  • interesting possibilities (Ewart). could even do an inotify-based reaction system based on filesystem changes

Erlang is hard to ship as it's a bunch of .beam files plus a VM. Go binaries are easy to deploy, well, 'cause they're one binary. (Riot artifact deployment cookbooks are what fujin uses for example)

Missing:

• Gohai
• Ewart - delegated subcomponents that could be written in different

Lessons learned from Go generally:

• Much less RAM usage & CPU.
• Other challenges however, but better tools -- memory, stack, heap, GC -- very well instrumented.
• Memory management is very different (C-like), but you have to account for memory leaks yourself. Ex you might write some code but you didn't realize you were creating additional heap objects every time.
• Diagnostic tools are a first-class citizen
• Ewart - pprof doesn't have enough resolution on OSX. WDYM, says fujin. Ewart - CPU has less resolution than memory. fujin: strengths are in GC instrumentation. can really get to a fine resolution.
• fujin: still challenged by composition 6 months later in Go. e.g. having to declare interfaces in advance.

Other implementations of Chef server?

• Ewart - has written a Chef server in Java using Dropwizard just for fun. Store data in memory or in Redis.
• Google - memcache "replacement" ("groupcache") - Brad Fitzpatrick et. al.
• John Ewart: John Keiser and him have been talking about a network of Chef servers (cache-hierarchy)

More on goiardi, and criticisms of push:

• shovey (Jeremy wrote it) instead of 'pushy' - without 0mq, uses Serf swimlane consensus protocol. almost compatible with knife-push
• Ranjib (and others) - doesn't know of anyone using Pushy.
• big criticism from all involved: Chef Push is all tied to a node attribute primitive. also, is it supposed to be an mcollective competitor? unsure about the use case
• shovey is tied to the goiardi server as a result, but Jeremy thinks that he'd like to break it out

• Serf is totally UDP-based

• Broker-based solutions like mcollective and pushy are not partition-safe (?)

• Q. Ewart: You want a primitive where you can block execution on something that does or doesn't exist globally? (quorum, mutex, etc.)

• A. Ranjib - Yep.

Some discussion of promise theory versus tell & fail.

Bunch of more discussion around distributing libraries as gems, etc. (Miah, etc.) Bunch of more digression about policy enforcement using Chef Client, etc.

LOL:

• Miah: Will it be awesome?
• fujin: No.
• Miah: Will it be better than Chef?
• fujin: Unlikely.
• Miah: You're not answering these questions correctly!
• fujin: Well look, it'll uncover some corner cases in Chef, and that might be interesting. And it provides a fast-prototype way of adding features.

Some digression about erchef storage format in PostgreSQL, where we compress JSON documents and store them. I pointed out that PostgreSQL 9.4 has much better JSON support in jsonb, so once that's released, it could be something we look at. Also allows you to do interesting things like index on JSON fields, do some JSON processing as SQL queries, etc...

What will we do now? What needs to happen next?