Agent Side functions in Puppet 6

Wait, that's a thing?

New talk, who dis?

profile

Ben Ford
Developer Advocate @ Puppet
@binford2k:

Hi, I'm Ben, aka binford2k on the interwebs.
I push the bits and talk to the peoples at Puppet and in our community.
If you've been around long enough, then I've probably told you to just fucking upgrade already at some point, so in keeping with tradition, I'll be doing that again today.
This time I'll do it by showing you a pretty rad new feature in Puppet 6 though.

Deferred Functions

people have been asking for them since...

...well, forever

Bits of code that can run agent side during catalog enforcement
Told in context of porting my own secret management module
Caveats, cautions, and guardrails

You didn't think I was going to tell you how to write Puppet code like a fancy shell script, did you?

Let's set the stage

puppetconf

Down in the bottom of that weird hotel, talking to people
I noticed a recurring theme throughout the whole event
Secrets, secrets, SECRETS!
Everyone wanted to talk secret management
Vault was like negative eight months old at this point

But with good reason. See, let's dig into a sneak peek into the innards of the Puppet Catalog.

Artist's Rendition of Puppet Catalogs

secrets everywhere

Let's level set first, because I don't want to scare you
Once Puppet's completed the initial certificate handshake, everything is SSL
So it's not like we're squirting secrets around Zune style
But the catalog itself has plain text secrets in it.
And there are reports, and diffs, and cached state files too.
The files are all protected by root filesystem permissions.
But that's a lot of eggs to put in one basket.
If an attacker gets ahold of the catalog, they've got the keys to everything.

Example Secret

Plain text in the catalog

file { '/etc/payroll/credentials.conf':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0600',
  content => 'some fake secret for cfgmgmtcamp',
}

bolt command run "cat /etc/payroll/credentials.conf"  --node agent

bolt command run "jq '.resources[] | select(.type == \"File\" and .title == \"/etc/payroll/credentials.conf\").parameters' /opt/puppetlabs/puppet/cache/client_data/catalog/agent.json" --node agent

Friendly disagreement

Is Puppet's CA sufficient for secret management?

Duty Calls -- someone is wrong

No surprise, in that loaded environment, I ended up debating secrets
The question was whether Puppet's existing CA and distribution mechanisms were more-or-less sufficient for basic secret management.
I thought that the agent could encrypt text using its certificate and the agent could then decrypt it using its own certificate
It would use the built in Puppet CA and meant that each agent could only decrypt its own secrets
Sure there were challenges, but I knew it would work conceptually.
But as often happens with strongly opinionated folk, neither of us changed position much.

So I'll just build it

We'll see if it works

ugh, fine. I'll do it myself

So some time afterwards, I just built the thing
- as one does
a secret management framework for Puppet with no external requirements
In a way, it was inspired by that friendly disagreement
But did it really work?
- Sure, it did the things -- thousands of people use it today
- But there were architectural challenges
- Let's explore those for a moment

Master-Agent Architecture

Puppet architecture

If you've taken one of my classes, you've probably seen this image enough to be sick of it.
The important part about this is that all processing happens on the master during compilation.
The catalog that gets sent back to the agent is just json... [switch]

Master-Agent Architecture

Puppet architecture

And that's important because it's just data.
It's nothing more that a list of resources and their properties.
There's nothing executable in there.
- with the obvious exception of embedded exec commands and the like
And this means that the agent doesn't run your code.
- All the Puppet code you've ever written has been run on the master.
Instead, the agent looks at this json state description and decides what tools to use to enforce that state.
This is how we make it declarative and immutable.

Example Catalog Resource

{
  "type": "File",
  "title": "/etc/puppetlabs/mcollective/server.cfg",
[...]
  "parameters": {
    "content": "plugin.activemq.pool.1.password = TekjzYZubjJG3WgSOkq4...",
    "owner": "root",
    "group": "wheel",
    "mode": "0660",
    "notify": "Service[mcollective]",
    "backup": false
  },
  "sensitive_parameters": ["content"]
},

For each resource
- Puppet chooses the best provider for that type
- Invokes the provider with the data you see here

Which means that only a provider can run code to decrypt the ciphertext

Which means...

At least one provider per type I wanted to support.

There were some 50 type in core Puppet at the time
And that's not even including modules, like concat or mysql

Encrypting a file in the catalog

How it works (3.x - 5.x)

A function run on the master during compilation:

Puppet::Parser::Functions::newfunction(:node_encrypt, [...]) do |args|
  content  = args.first
  certname = self.lookupvar('clientcert')
  Puppet_X::Binford2k::NodeEncrypt.encrypt(content, certname)
end

And a provider run on the agent during enforcement:

Puppet::Type.type(:node_encrypted_file).provide(:ruby) do
  desc 'Ruby provider for the node_encrypted_file type'
  # ...

  def content
    Puppet_X::Binford2k::NodeEncrypt::Value.new(File.read(resource[:path]))
  end

  def content=(value)
    File.write(resource[:path], resource[:content].decrypted_value)
  end
end

That was a lot of providers, so .... I only ever got around to writing one
But this is how it works
It was a provider for an encrypted file
the only thing it could manage was file content
decrypted using the agent's certificates

Packaged up in a defined type

define node_encrypt::file (
  $ensure                  = 'file',
  $path                    = $title,
  $backup                  = undef,
  $checksum                = undef,
  $content                 = undef, # managed by the custom provider
  $force                   = undef,
  #...
) {
  file { $title:
    ensure                => $ensure,
    path                  => $path,
    backup                => $backup,
    checksum              => $checksum,
    force                 => $force,
    #...
  }

  node_encrypted_file { $path:
    content => $content,
    before  => File[$title], # let the File resource do all the work
  }
  Node_encrypt::File<| title == $title |> {content => '<<encrypted>>'}
}

As you can see, while it worked fine, it was tedious as hell.
I wrapped up a file resource in this defined type instead of doing it in the Ruby provider so that I didn't have to make two providers inheriting from each of posix and windows.

Example Secret with Stub File

Wrapper type encrypts content

node_encrypt::file { '/etc/payroll/credentials2.conf':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0600',
  content => 'some fake secret for cfgmgmtcamp',
}

bolt command run "cat /etc/payroll/credentials2.conf"  --node agent

bolt command run "jq '.resources[] | select(.type == \"File\" and .title == \"/etc/payroll/credentials2.conf\").parameters' /opt/puppetlabs/puppet/cache/client_data/catalog/agent.json" --node agent

bolt command run "jq '.resources[] | select(.type == \"Node_encrypted_file\" and .title == \"/etc/payroll/credentials2.conf\").parameters' /opt/puppetlabs/puppet/cache/client_data/catalog/agent.json" --node agent

What about every other resource type?

think about

This would be so much more useful if I could just run a function on the agent during enforcement...

Sure enough, almost immediately someone wanted to encrypt a user password on Windows
And then someone else wanted to use concat with a secret
And I couldn't support either, without writing custom providers for each.

Puppet 6 introduces Deferred Functions

Evaluated by agent during enforcement

Building a template on the agent:

$variables = {
  'password' => Deferred('vault_lookup::lookup',
                  ["secret/test", 'https://vault.docker:8200']),
}

# compile the template source into the catalog
file { '/etc/secrets.conf':
  ensure  => file,
  content => Deferred('inline_epp',
               [file('mymodule/secrets.conf.epp'), $variables]),
}

Any Puppet 4.x function that returns a value can be deferred except for those using scope variables or interacting with the catalog.

Secret value is resolved on agent
Template is rendered on agent
Declare an instance of the Deferred data type
Give it the name of the function you want to call and an array of arguments
Use inline_template() and file() to include template source in catalog
You can defer most Puppet 4.x functions
Note: only .epp style templates can be deferred due to how ERB uses scope.

But hold up a moment!

Deferred functions are designed very specifically

There are guardrails in place to ensure that deferred functions continue to be part of a maintainable and declarative codebase.

Some implementation details that will matter:

Deferred is a data type, so it returns an Object.
Deferred functions must resolve to a value.

We will revisit these points

This sounds like I'm talking about breakfast cereals here
But these are important points for keeping your code manageable long term.
I spend a lot of time in Slack rescuing people from bad decisions.
Selfishly, I don't want you to be one of them.

Porting the module

From custom providers to deferred functions

After all that buildup, I'm afraid this might be a little anticlimactic.

Do you remember what's needed to defer a function?

Porting the module

From custom providers to deferred functions

After all that buildup, I'm afraid this might be a little anticlimactic.

Do you remember what's needed to defer a function?
Just expose the decrypt half as a Puppet 4.x function:

Puppet::Functions.create_function(:node_decrypt) do
  dispatch :decrypt do
    param 'String', :content
  end

  def decrypt(content)
    Puppet::Pops::Types::PSensitiveType::Sensitive.new(
      Puppet_X::Binford2k::NodeEncrypt.decrypt(content)
    )
  end
end

Example Encrypted Secret

Deferring the function

file { '/etc/payroll/credentials3.conf':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0600',
  content => Deferred("node_decrypt",
                [node_encrypt('some fake secret for cfgmgmtcamp')]
             ),
}

bolt command run "cat /etc/payroll/credentials3.conf"  --node agent

bolt command run "jq '.resources[] | select(.type == \"File\" and .title == \"/etc/payroll/credentials3.conf\").parameters' /opt/puppetlabs/puppet/cache/client_data/catalog/agent.json" --node agent

There you have it.
All you need to defer a function to agent enforcement time is a Puppet 4.x function that doesn't do any funny stuff.
It's even relatively straightforward to use.
Use node_encrypt to encrypt on the master, then defer node_decrypt to the agent
.....but that's a lot of typing, yeah?
We can do better

Deferred is a Data Type

A value expression

Returns a callable object
Wraps a function call and its arguments
Assign to variable like any other value expression
... or return it from a function:

function node_encrypt::secret(String $data) >> Deferred {
  Deferred("node_decrypt", [node_encrypt($data)])
}

Remember how I said that Deferred is a data type?
This is why that's important.
The thing it returns is just an object that you can do things to like any other object.
In our very first example, with the agent side template, we assigned it to a variable.
But we can also return it from a function.
So this is a Puppet language function
Which means that I can basically just cut & paste the code from the last slide in

User friendly function

Deferred secret wrapped by a function call

Just tag the string:

file { '/etc/payroll/credentials4.conf':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0600',
  content => 'some fake secret for cfgmgmtcamp'.node_encrypt::secret,
}

bolt command run "cat /etc/payroll/credentials4.conf"  --node agent

bolt command run "jq '.resources[] | select(.type == \"File\" and .title == \"/etc/payroll/credentials4.conf\").parameters' /opt/puppetlabs/puppet/cache/client_data/catalog/agent.json" --node agent

Yes, that's it.
It only took a few minutes to read the docs and do the port.
But now we can tag any string in our codebase and it's cast immediately to an encrypted blob of ciphertext that's automatically decrypted on the agent at runtime.
I think that's pretty rad and it was shockingly easy to do.

Deferring functions is easy

But let's talk about when you should do it

You were so preoccupied...

Remember that I said this was going to be a lot about guardrails
And that we weren't going to be talking about how to use Puppet as a fancy shell script engine
This is when we come back to the other point I mentioned earlier.
The only functions it makes sense to defer are those which return a value.
This framework does not allow you to defer logic
You cannot make decisions based on an agent side function
All it allows you to do is resolve a value as the catalog is enforced.
That's it; resolve a value.
And there's reason for this.

Catalogs represent state

Describe all configuration you care about

Determinate
Repeatable
Disposable

toss test

If you cannot dispose of any given server and rebuild it trivially, then your configuration is insufficient.

I probably don't need to say this in 2019, but your catalog should describe everything that matters about this server.
The hardware--or virtual hardware, container, etc--it runs on shouldn't matter
The server is just the executable form of that configuration model
Which means that your model should be simple and descriptive; not a fragile shell script.

Known Unknowns

aka, resolving values at runtime

We already do this all the time!
Items where the specific value is less important than what it is.
- The latest version of PostgreSQL
- Whatever address mongo.example.com resolves to
- The master branch of the git repo
- Admin user accounts from LDAP
- Admin password from Vault
Things where a placeholder that resolves at runtime is an acceptable risk.

To be sure, these often do cause problems themselves, but that's outweighed by the development velocity they enable and the internal consistency they can provide (when they're not broken),

But the computing world has a long history of using the known unknowns--like variables.
Facts are resolved early in the compilation, so we can just use their values.
We only put late bound variables in the catalog when being resolved at runtime is preferable.
I like to think of these late-bound variables as labels, because they're just a named placeholder for data
And importantly, the name describes what they are, which is often more important than what their value is
for example, that mongodb.example.com address is actually more specific than if we put the IP address
- because when we update DNS, all the hosts talking to it just use that new address.
- so the domain name has specific meaning as "whatever address this currently resolves to" instead of just an IP address that was correct at some point in time.

Deferred function use cases

Only when you need a placeholder for a value

Agent side functions should not make config changes, or have side effects, or attempt to be conditionals. They should only resolve a value at runtime that should only be known by the agent.

Most system info values should continue to be facts
- Allows info to be surfaced in PuppetDB too, for example
Use deferred functions when:
- Secret values that should not be known by the master
- Want values to update on each run even when using cached catalogs
- Agent has access to data master doesn't and you can't mirror
...and maybe a few edge cases

Example Use Cases

Things that make sense to use Deferred functions for

Secret management:
- Retrieving secrets from a secret server
- Decrypting values agent side (binford2k/node_encrypt)
Service discovery:
- Consul (ploperations/consul_data)
- Zookeeper
- etcd
Reduce load on the master by offloading a heavy computation:
- Generate self-signed SSL certificates for test webservers
Make an API call:
- Register in some kind of directory service
- Trigger completion of a step in your provisioner

See, I'm already violating the rules I jsut gave you.
- Because the act of making an API call is depending on a side effect.
- That's important, because intent, clarity, and maintainability take precedence.
node_encrypt is the module I just ported for you
consul_data is a newly developed module by our internal ops SRE team for declarative agent side service discovery.
You're invited to try them out, offer feedback, submit PRs, etc.

Immutable Configuration

For all practical purposes

Puppet converges to a defined state every 30 minutes by default
Long term, the state you define is the state your node will be in
It's up to you to make that state definition as descriptive as possible
- Use placeholders for data when that makes sense
- Because that label describes intent more than the value it resolves to
Your model is more repeatable when you depend on fewer external artifacts

Ideally you want each node itself to be utterly irrelevant because you can trivially build a new one configured exactly as needed.

And this gets us to a more-or-less immutable configuration.
Puppet will converge you to defined state each time it runs
- So long-term, your configuration is stable at that defined state.

Summary

Agent side functions are totally a thing now.
Any Puppet 4.x function can be deferred
- Resolve a value
- Has no side effects
Use deferred functions as placeholders for late-resolved values
Should focus on providing intent

Contributor Summit: 2019

Budapest in June!

And this is my attempt at a markety slide! Do you like it?

Come see me in Budapest and learn more about deferred functions and other cool new tech!
We'll be running another Project Month building up to it
I'd love to see some projects impementing deferred functions
- Or Bolt backed projects
- Or even something using Lyra
Keep an eye out, we'll be announcing details soon.

fin

Deferred functions in Puppet 6

Ben Ford
Developer Advocate @ Puppet
I build things and talk to people
@binford2k:

And that's it.
Again, I'm Ben Ford--or binford2k on the internets--and I work at Puppet.
I'm our Developer Advocate, which means that I get to build cool shit and talk to people.
And that makes me happy, which is all we can ever ask for, right?
Thanks for listening.

Showoff Menu

Agent Side functions in Puppet 6

Wait, that's a thing?

New talk, who dis?

Deferred Functions

people have been asking for them since...

...well, forever

Let's set the stage

Artist's Rendition of Puppet Catalogs

Example Secret

Plain text in the catalog

Friendly disagreement

Is Puppet's CA sufficient for secret management?

So I'll just build it

We'll see if it works

Master-Agent Architecture

Master-Agent Architecture

Example Catalog Resource

Which means...

At least one provider per type I wanted to support.

Encrypting a file in the catalog

How it works (3.x - 5.x)

Packaged up in a defined type

Example Secret with Stub File

Wrapper type encrypts content

What about every other resource type?

Puppet 6 introduces Deferred Functions

Evaluated by agent during enforcement

But hold up a moment!

Deferred functions are designed very specifically

We will revisit these points

Porting the module

From custom providers to deferred functions

Porting the module

From custom providers to deferred functions

Example Encrypted Secret

Deferring the function

Deferred is a Data Type

A value expression

User friendly function

Deferred secret wrapped by a function call

Deferring functions is easy

But let's talk about when you should do it

Catalogs represent state

Describe all configuration you care about

Known Unknowns

aka, resolving values at runtime

Deferred function use cases

Only when you need a placeholder for a value

Example Use Cases

Things that make sense to use Deferred functions for

Immutable Configuration

For all practical purposes

Summary

More Infos

Docs are in the Custom Functions section

Contributor Summit: 2019

Budapest in June!

fin

Deferred functions in Puppet 6