Thusrday, Issaquah, 10:30am

Justin Redd

Heph, Charles Johnson, Justin Redd, Joseph Holstrom, Jesse, dozens of others

"When nodes are changing names during

Justin: How to deal with machines changing IPs or names? Heph: We use Chef Search for service discovery. Not ideal because there's heavy delay. Starting some work moving into Consul-based service discovery. REALLY NEAT because it has lots of the things built into it that like smartstack and etcd don't have, which is built-in service monitoring. Can automatically remove or add selves to pool based on service status. Since currently everything that's doing service discovery through Chef is doing erb templates, have an ohai plugin to Consul to get a list of every server. Get it as node attributes instead of Chef Search attributes.

Q: How are nodes told to reconfigure? A: Not ideal at the moment.

Q: Do you use Consul's watch functionality? A: Not right now.

Comment: They just shipped watches as a top-level function in Consul 0.4.

Question: Can someone give a quick summary of Consul? A: C-O-N-S-U-L. From Hashicorp, the Vagrant cats. It's a gossip-based service discovery protocol that also has a cluster of consistent servers that have the final vote for everything. Run a daemon on every host with a set of service checks. If checks good, report back to cluster and advertise. Can be easily queried via HTTP interface, but also has a DNS interface. So right now nginx boxes are pointed at a whole stack of Python app servers with Python.service.consul as upstream DNS, dynamically resolving on every request. Automatic round-robin DNS on 100 nodes, and Consul immediately handles group membership. Also has a k/v store in the HTTP API. Apparently it has Watchers now built in.

Q: And what are watchers?
A: Set subscribers on keys, if they modify then agents will take action. Every node runs a client, all participate in gossip, so every request just happens to loopback.

Q: How many nodes on Consul?
A: About 400 so far.

Q: What about DNSSec?
A: Have to disable DNSSec to use it.

Comment: It's not really a consistent read out of the DNS interface. Can do a strong read over the HTTP API.
A: You CAN do a strong read off of DNS, it's an option.
C: Neat!
A: You can ask for stale data from local node instead of waiting for a true lookup. Diminishes big reads. C: Also has the idea of data centers built-in, can do multiple networks / wide-area.

A: Think of it like SRV records, gives you checks to do inclusions or exclusions based on checks.

How to integrate with Chef? A: Recipe lays down role that sets a check.

Q: Are there consistency problems between Chef and Consul? A: All the time. Chef Search says that something is up, and Consul says it's newer.

"It's monitoring 2.0 grade."

Comment: "There's an RFC out right now that's about attributes, building a new concept of node state where we might be able to use consul or etcd or zookeeper. I'd really - Ranjib & I were talking about prototyping a provider that would allow you to do this. Take advantage of near-realtime K/V stories that are strongly consistent vs. what the Chef Server is doing. If you're using etcd or consul AND you're using Search, then you're doing it wrong. Have one source of state for this."

"So with Chef Server and Consul you have conflicting consistency models."

Comment: "Chef server says there's a node out there that may or may not be workable right now. Consul says there's a node out there right now that is available."

The Lambert clock: What happened first, what happened next.

Let's back up and overview. What are the options?

• Consul (hashicorp)
• Zookeeper (Apache)
• ETCD (CoreOS)
• Smartstack
• Chef Search

Zookeeper and etcd run on the notion of reported state with timeouts. "I am an app server, and I'm healthy." With a 300sec timeout. Zookeeper requires a constant TCP session.

Any solution requires code changes.

Q: "What needs to happen in the actual application?" A: "For any of these you have to do something in your application to say 'I am publishing a record based on some state.' Within clients you have to do something to notice when that state changes and take appropriate action." A: "Zookeeper and etcd have actions. etcd is a rest service, zookeeper is a binary-" A: "Most of these are coming from Zookeeper as ideas, but the implementations are very different. Zookeeper uses Paxos and is very chatty. Traffic increases as network increases. etcd is built on RAFT, works over flaky networks. Serf came with gossip for HA, and then Hashi took Serf + RAFT into Consul." A: "There's a book, distributed systems for fun and profit. Short and easy."

A: "You guys are in a rabbit hole. These are all storage engines. The consistency model of your storage engine isn't the biggest problem you have to tackle. Let's assume that it's fine. We can have an academic discussion about linearization later. The real problem is integration with existing apps: Getting up-to-date data into your storage engine, and notifying applications when state changes and getting data out."

A: "I think you've gone too far. When you start considering self-healing apps like you're alluding to, you have to change the application. If there's an operator who can know a system is unhealthy and I can take a manual step to fix it, then that's maybe okay. Later I can work on self-healing. There's an evolution before revolution."

A: "Current problem plaguing Chef users is the storage engine."

A: "Don't use Chef for service discovery." (ARGUMENT INTENSIFIES) "If all you're doing is provisioning 10 nodes a day, why not?"

A: "The barest simplest implementation that anyone can do is Chef Search. It's slow, not even close to real-time, high cache interval."

A: "It got faster with Chef Server 12." A: But query lag isn't the issue, it's that the object isn't updated until the client runs.

Back to Consul: "Consul can use service checks based on Sensu."

Q: Integrations with Chef handlers? A: (Ranjib): "Host discovery isn't service discovery. You want is Tomcat up? Then you have to model service discovery." (Discussion runs out as to whether you can do service checks in consul or not.) "Zookeeper is flaky over WAN. It's discovery, someone has to consume the discovery as well. But not everybody needs it, and it's a complex thing. Not everybody needs it."

Q: How do you know you need it? A: If you have lots of node churn, or if you're deploying and want to advertise, use it. If you don't, if you're just provisioning apps with Chef and then using middleware with its own service discovery, maybe you don't need it.

"I use it for load balancers. Pub/sub."

Q: How do you deal with nodes getting pulled out of the pool? Is it self-healing? A: Self-healing is - heh. What I'm doing right now is - an nginx pool that routes traffic to back-end worker hosts, via DNS lookups. If it goes away we just remove it from DNS. Q: So every time it's trying to proxy outbound, it does DNS against a local outbound? A: Yep. Q: Cacheing? A: I'm having issues. Trying to figure out how to put BIND in front of it.
Q: Does cache exceed the priority for the stale data? If you do a BIND cache - the result is that during that cache lifetime, the node is down, and you're in TTL A: Each nginx box terminates 2500 connections/sec, that's a LOT of DNS lookups for this little Go process. I want whatever I do to require minimal code changes, my dev team isn't as responsive to ops requests as product requests. C: "But, but Devops!"

Q: "Also these are high-velocity projects. Lots of changes and breaks." A: "Consul is changing, etcd is changing. Zookeeper is stable, but zookeeper is JVM and stable." A: "Twitter uses Zookeeper for everything. Massive, fucking big infrastructure. But we had a mess of teams dedicated to the next phase, we were gonna hit the upper limit because it will saturate the network." A: "Zookeeper is the Cadillac of consistency systems. Most robust but not most resilient. It's a CP system, if anything goes wrong it stops."

Q: "Question that's orthoganal to everything. We're talking about service discovery, interesting for finding hostname, IP, port. But what do you do for database clients, when you need to distribute a security credential to create a connection?" A: Exasperated laugh. Q: "You're passing out things to create endpoints, but not enough to create connections." A: "Secure credentials in the cloud is its own room. Having secure credentials per host makes it exponentially more complex. All of these could let you attach metadata to a port, so if you wanted to do it wrong you could distribute it that way." A: "We're using separate systems for that."

Q: We have copies in multiple places, one in Chef Server and one in (whatever). What if we think about the Chef Server as one engine, and then we plug in different engines? A: There's an RFC! There's an OHAI plugin to Consul, get the contents of the datastore into your node data. A: Ranjib & I did something similar, our thought was - Let's destroy the search class, build a whole new one with our own back-end. Search is SOLR syntax, so - getting your kv store to handle the search syntax is a pain in the ass. A: Ranjib says: "We had this discussion last year, discussed different solutions. If we wanted an API in Chef, with pluggable back-ends." A: Joseph: "I've been working on that since last year. The question is, can you unify the query API? I think that's insane and not worth solving." A: Daniel wants to do things with attributes. Node.set is a data store - persisting data into the Chef Server. People do that, then they try to build these things. In my strong opinion wen eed to bring in an abstraction over a CP system - could be a REDIS instance at the most simple form, at the most complex it's a giant zookeeper / consul kv store. And then have primitives in the Chef DSL that allow you to do strong reads against a value, or persist some node state data. Destroy the idea that in the converge cycle there is data that isn't persisted. Get rid of the idea of the attribute merge. Build the primitive in the DSL that gives you this. From there you can build sequencing locking tasks. So what does that API look like?"

Q: "But I'm looking for push-based notification, not pull-based search. So bringing the Chef Server into this seems like unnecessary complexity. There are other systems out there to do it better."

A: My answer is less about that, more about - I need to know what version to put where? So there are 2 things to happen: I want to put that data somewhere, and then I need to trigger the run.

Q: But what if we need it as close to now as possible?

A: With consul or etcd with confd it can watch the datastore, can trigger actions after particular data changes.

Q: "Joseph: So Confd is extremely opinionated. I'm trying to build something - The reason I've been building this project called discoteq is to build an unopinionated version of confd. I've built a Ruby and a go implementation. ruby embeds into cookbooks just fine. Been meaning to take the elasticsearch cookbook and translate its old search thing to this new API to prove that it can work, but it's still not got support for consul/zookeeper. Only has support for etcd and zookeeper at the moment. It's on Github. Basically I will go dig into whatever you want it to integrate with at the moment, recently went and fixed the tilt gem to read from JSON files so you can use any template engine supported by tilt to hook up to this thing.

Q: "Bringing it around: Justin is why we're sitting here, about service discovery. What do you think?" A: "Why do I care about consistency? What do I care if my node gets the same answer as the one next to me, or the next read?" A: "Clusters, Cassandra for instance."

A: What if we can just bring one attribute from one node straight from the db instead of search?

Q: What about routing to broken nodes? A: One of the best implementations of fixing that is Smartstack. Not writing an actual config file down anywhere.

Q: So what are you doing with these consistent datastores when something goes away? A: Reconfigure things that need it as a response. Q: So this doesn't work unless your application supports really lightweight reloads? A: From a golang example, close outbound connections and re-open if they're using config data from the k/v system. Q: So if you have a legacy app that can't hot reload, are you insane to - A: Use Confd. Q: So you're sticking an eventually consistent datastore on top of your consistent datastore. You're not making a db query with every request you're responding to. A: But it's still eventually consistent. Q: For legacy applications, we are at least going to get the 30 second delay of updating haproxy or running confd. Is that okay? Do we really need to have a CP store instead of an AP store? A: If a service goes away and you're looking it up for a load balancer or something with a built-in health check, that's fine. Haproxy sends a 1s health check to my upstreams anyway. Adding new ones does require a config reload, can't do it via the socket/live. Point being we have to do restarts. If app is aware of config store and can query in realtime, then that's awesome. But most apps aren't aware, because this is new.

• Start an RFC: Describe an API for primitives. Is that Discoteq? Pluggable Chef Query is what we want. Backend-agnostic Chef Server Query Interface. An endpoint for strong reads, possibly side-by-side with Search. Tie into other things.
• Release of OHAI plugin for consul, if possible.