Showoff Menu

Table of Contents

Press ? for help.

All features are anonymous.

loading presentation...

Sync Presentation

Telemetry doesn't have to be scary

Developer focused metrics that make sense.

Let's go ahead and get started today
I'll be talking about telemetry, also known as kryptonite to many of us.
But really it's how we know what to work on!
so today, I want to talk about metrics focused on helping YOU.

New talk, who dis?

profile

Ben Ford
Developer Advocate @ Puppet
@binford2k:

Hi, I'm Ben, aka binford2k on the interwebs.
I push the bits and talk to the peoples at Puppet and in our community.
If you hang out in our Slack, you've probably seen me grouch that not knowing how people use Puppet holds us back from improvements.
So in keeping with tradition, I'll be doing that again today
But this time I hope to have a solution.

Why we need metrics

Photo by https://unsplash.com/@benknight17

Because that's how you know how the tools are being used.
And not knowing this information is like flying blind.
Off into the great white unknown.

Photo by https://unsplash.com/@benknight17

I love the PDK

It's not a huge task to design and build a simple, single-purpose, bespoke Puppet module in a short period of time.

Let's start at the beginning.
Identify a problem you want to solve.
Create a module.
Dev tools streamline the process.
- pdk new module
- pdk new class
Write a little Puppet code.
- VScode and Puppet plugin
Even get autogenerated testing and linting
- pdk validate
- pdk test unit
So what's the problem?

Maintenance is a different story

Hey, can we add ArchLinux support?
What about Windows 2012?
Here's a PR for a defined type to manage the database too... ^{(MariaDB only)}

It's really really easy to scope creep.
Accepting a PR is trivial. Maintaining it may not be.
Every added functionality you add is more complexity to maintain.

Development resources

Every single platform or feature you support takes time to maintain it!

To make the most effective use of our time, we should focus on the most used code
If people aren't using a class, then dedicating maintenance time to it is wasted
Especially if that code adds complications
- fragile testing
- more choices lead to complex user experience
- unneccesary abstractions make code harder to read
- I have an old Drupal module myself that includes sub-classes based on a complex case statement and string interpolation and why?
I remember reading an article once that explains how your business should "fire" customers that cost you more to keep than they actually generated.
Likewise, sometimes we need to "fire" code that's not worth keeping.
But how do we know? How do we make these decisions?

Data driven development decisions

two roads diverge

Two roads diverged in a wood, and I—
I took the one less traveled by,
And that has made all the difference.
-- Robert Frost

Photo by https://unsplash.com/@leliejens

With all due respect to Robert Frost; as developers, we generally want to take the road MOST traveled!
Just like seeing the wear patterns in these roads, knowing the usage patterns of our code is how we know what code to focus on.
Or even what code to dump.
Saas applications are used to easy access to profiling metrics because most platforms have endpoint tracing built in.
Twitter always knows what parts of their platform are used most consistently.
For on-prem software like Puppet modules though, we have to gather those patterns in other ways.

Photo by https://unsplash.com/@leliejens

Impact Analysis

Photo by https://unsplash.com/@the_roaming_platypus

let's start out by talking about Impact Analysis
this basically means identifying the effects of changes we make
it's most effective when you can do that before breaking things!

Photo by https://unsplash.com/@the_roaming_platypus

Rangefinder

[~/Projects/puppetlabs-concat]$ rangefinder manifests/fragment.pp
[concat::fragment] is a _type_
==================================
The enclosing module is declared in 173 of 575 indexed public Puppetfiles

Breaking changes to this file WILL impact these modules:
  * nightfly-ssh_keys (https://github.com/nightfly19/puppet-ssh_keys.git)
  * viirya-mit_krb5 (git://github.com/viirya/puppet-mit_krb5.git)
  * rjpearce-opendkim (https://github.com/rjpearce/puppet-opendkim)
  * shadow-tor (git://github.com/LeShadow/puppet-tor.git)
[...]

Breaking changes to this file MAY impact these modules:
  * empi89-quagga (UNKNOWN)
  * unyonsys-keepalived (UNKNOWN)
  * Flameeyes-udevnet (UNKNOWN)
  * ricbra-ratbox (git://github.com/ricbra/puppet-ratbox.git)
[...]

Identify the public Forge modules using parts of your code.

https://github.com/puppetlabs/puppet-community-rangefinder

Rangefinder is a tool I've been working on for a while.
You just run it on a file and it figures out what that file defines and then tells you what Forge modules use it.
In other words, if you break that file, who will be impacted?
If you run it against a whole module, it iterates everything in the module.

Run it on a pull request!

Rangefinder webhook

Rangefinder also comes in a GitHub App flavor.

https://github.com/apps/puppet-community-rangefinder

https://github.com/apps/puppet-community-rangefinder

Static analysis of dependencies

network

Works by dissecting latest releases of all modules on the Forge
Static code analysis identifies what every module uses
- resource types
- functions
- classes
Runs weekly to generate dependency network

This uses my itemize library: https://github.com/binford2k/binford2k-itemize

Photo by https://unsplash.com/@alinnnaaaa

This uses my itemize library: https://github.com/binford2k/binford2k-itemize

Photo by https://unsplash.com/@alinnnaaaa

Dependency data is public

BigQuery console

The data Rangefinder uses is stored in a public BigQuery database.

Find more information about how it works and other cool queries you can run on the dataset powering it at my blog: https://binford2k.com/2020/04/06/rangefinder/

You'll find a lot more information about the tool and the dataset powering it at my blog, binford2k.com.

Major limitation

Public market

Rangefinder only works on public Forge modules (currently).
Cannot identify how code is used by profiles or private modules.

Photo by https://unsplash.com/@sguestsmith

Right now, Rangefinder can only identify dependencies from public Forge modules.
To truly know the spread and the impact of breaking changes we need production usage patterns from infrastructures actually using modules and components.

Photo by https://unsplash.com/@sguestsmith

Privacy. It's a thing.

Photo by https://unsplash.com/@pawel_czerwinski

But that immediately raises the privacy red flag.
Any time a tool I use phones home, I immediately wonder what it's gathering.
Whether it's sneaking information that I don't really want to share.
I run Little Snitch, just like many of you, and I often even bust out Wireshark.
So privacy and maintaining trust was the first thing on my mind.

Photo by https://unsplash.com/@pawel_czerwinski

Business Identifiable Information

telephone

Sharing too much implementation information about how an infrastructure works is just asking for security compromises.

Most of you know BII's big cousin, PII.
But many of the same considerations apply to businesses.
Many companies don't allow phone-home metrics at all.

Do I really want people knowing the name of my primary LDAP server?

Photo by https://unsplash.com/@quinoal

Fingerprinting

fingerprint

A more insidious danger is fingerprinting.
Identifying individuals by properties or habits.
Also applies to infrastructures.

If you saw that a site was in the CEST time zone, had the locale set to fr-ch, and had thousands of nodes classified with various HPC modules, then you might guess that you were looking at CERN.

For example, if you saw that a site was:

in the CEST time zone (central european summer time)
had the locale set to fr-ch (swiss-french)
had tens or hundreds of thousands of nodes classified with various HPC modules

then you might surmise that you were looking at CERN and use the rest of that record as reconnaissance against them!

Live example of fingerprinting

http://panopticlick.eff.org/

Hard requirements

Provide real value to the end user. ^{(that's you)}
Transparent about what data is collected, why, and what it's used for.
Give you the ability to opt in or out without gating other benefits.
The data collected can be aggregated to share publicly.

When I started designing the framework, I set out four invariant criteria.
Much of it is to ensure that the act of sharing metrics data provides the user with value.
In other words, I don't want to incentivize you to share metrics because sharing your metrics should provided you with value!

Telemetry Data Model

Photo by https://unsplash.com/@crissyjarvis

So let's look at the data model.
And show you how we are meeting those requirements.

Photo by https://unsplash.com/@crissyjarvis

Public dataset

No access to individual records themselves

-- The top ten most used classes in the ecosystem
SELECT name, count
FROM `dataops-puppet-public-data.aggregated.class_usage_count`
ORDER BY count DESC
LIMIT 10

[
  { "name": "Resource_api::Agent", "count": "272" },
  { "name": "Account",             "count": "272" },
  { "name": "Ssl::Params",         "count": "272" },
  { "name": "Classification",      "count": "272" },
  { "name": "Os_patching",         "count": "269" },
  { "name": "Ntp",                 "count": "265" },
  { "name": "Ntp::Install",        "count": "265" },
  { "name": "Ntp::Config",         "count": "265" },
  { "name": "Ntp::Service",        "count": "265" },
  { "name": "Zsh::Params",         "count": "265" }
]

You'll need a Google Cloud account and then you can access the dataset with your browser via the BigQuery Console. Then you can run any queries you'd like.

Let's start from the end here
This is the data that you (or anyone else) has access to.
All you need is a Google Cloud account.
This query shows the ten most used classes currently in the dataset.
Note that you cannot see any actual records themselves, so you cannot see the complete set of classes that any site is using.
All you see is a lot of pre-computed sums, counts, and other data aggregations.
The actual data is currently super limited right now -- only alpha/beta testers have reported in.

Aggregating data

Data is generated via SQL queries:

-- Generate an aggregate table of all classes used and their counts.
SELECT classes.name, classes.count
FROM `bto-dataops-datalake-prod.dujour.community_metrics`,
    UNNEST(classes) as classes
GROUP BY classes.name, classes.count

aggregation

https://github.com/puppetlabs/dropsonde-aggregation

Each week, a bunch of little SQL queries run to generate that data.
They take the real data and strip anything identifiable.
Count, sum, etc. and generate metrics ready to be used.
Every query generates a table.
There are not a lot yet, but we'll be adding them as we learn useful ways to dissect the data.
In case you've been admiring the wonderful imagery in this deck up until now, yes, this OmniGraffle thing is how sophisticated my own images are.

Private dataset

So about that sensitive data...

No trespassing

Telemetry reports are not terribly sensitive, but they are fingerprintable.
So we store them only in a secured dataset.
Limited access, only myself and our BigQuery admin team.

Our own developers who want to use collected metrics data have to do it via aggregation queries just like you would.

Metric Plugin Design

Photo by https://unsplash.com/@danielkcheung

Now we'll talk about how the telemetry framework itself protects your privacy.
Metrics are each designed as self-contained plugins.
And every time they run, they're validated for consistency.

Photo by https://unsplash.com/@danielkcheung

Plugin hook design

Plugins are implemented as hooks.
- description
- schema
- example
- run
Some hooks are optional
- initialize
- setup
- cleanup

description describes what the plugin does
schema describes the data that the plugin returns
example provides a representative example of what the metric data could look like.
run actually evaluates the metric and returns real data
the optional hooks, initialize/setup/cleanup, just provide ways to separate out the code into cleaner stages

Schema hook

Describes data that the plugin is allowed to return.

Returns a BigQuery schema.
JSON format with an array of row definitions

def self.schema
  [
    {
      "description": "The number of environments",
      "mode": "NULLABLE",
      "name": "environment_count",
      "type": "INTEGER"
    }
  ]
end

See full schema of all plugins with dropsonde dev schema

Run hook

Generates and returns data for the metric

Returns an array of rows of data.
Can return any data that the Puppet Server node can generate.
Use PuppetDB, codebase, etc.
Methods provided to filter out private modules.
Must match schema or plugin fails.

def self.run
  [
    :environment_count => Puppet.lookup(:environments).list.count,
  ]
end

Example hook

Representative example of sample data

Provide one element of randomized representative data.
Used to generate a dataset that's shaped like the real dataset.
Must match the schema or plugin fails.
Used for developers to generate aggregation queries.

def self.example
  [
    :environment_count => rand(1..100),
  ]
end

Available as dataops-puppet-public-data:community.community_metrics

Available as dataops-puppet-public-data:community.community_metrics

Previewing a telemetry report

$ dropsonde --enable environments preview

                      Puppet Telemetry Report Preview
                      ===============================

Dropsonde::Metrics::Environments
-------------------------------
This group of metrics gathers information about environments.
- environment_count: The number of environments
    4


Site ID:
bab5a61cb7af7d37fa65cb1ad97b2495b4bdbc85bac7b4f9ca6932c8cd9038dd7e87be13abb367e124bfdda2de14949f8b3e8658931e39f58bcef23382d8f426

Before you submit data, you can see what's generated
This example selects only a single metric
Site ID is a cryptographic hash that just correlates reports
- Can be regenerated by providing a seed value

Data validation

halt!

The plugin cannot return data that it does not declare.
The database schema is also defined from the plugin schemas.

Using the Data

Photo by https://unsplash.com/@sonson

Public data

dataset screenshot

All you need is a Google Cloud account.
Then you can explore the datasets.

Once you have an account, browse to https://console.cloud.google.com/bigquery?p=dataops-puppet-public-data&d=aggregated to check out the dataops-puppet-public-data project.

Interesting queries

Dataset includes:
- Forge module metadata
- Static analysis of Forge modules
- GitHub repositories that look like Puppet modules
- Aggregated telemetry data
Combine them as you will.

Example: a list of the modules on the Forge that define custom native types:

SELECT DISTINCT g.repo_name, f.slug
FROM `dataops-puppet-public-data.community.github_ruby_files` g
JOIN `dataops-puppet-public-data.community.forge_modules` f
    ON g.repo_name = REGEXP_EXTRACT(f.source, r'^(?:https?:\/\/github.com\/)?(.*?)(?:.git)?$')
WHERE STARTS_WITH(g.path, 'lib/puppet/type')
LIMIT 1000

So what else is possible?

Who knows, really.

Maybe we can incorporate Impact Analysis into PDK validations.
Maybe the Forge will start surfacing some of these metrics. ^{(hint: it will)}
Maybe Vox Pupuli will incorporate usage data into their Tasks management interface.
Maybe you will write something I haven't even thought of.

Vox Pupuli Tasks interface

Get Involved!

Photo by https://unsplash.com/@lunarts

Like any good Community project, this doesn't actually go anywhere until I get you all convinced.
And honestly, it totally depends on at least some of you getting interested.

Photo by https://unsplash.com/@lunarts

What I'm asking from you

This is totally opt-in only right now.
puppet module install puppetlabs/dropsonde
Check out the example dataset and contribute some aggregation queries.
Develop and contribute a metric plugin.
Doesn't have to be strictly Puppet, just ecosystem:
- Reid already made a plugin to see the impact of building a better control repo and Puppetfile for better r10k deploys.
- David could see how many people are using the Rocket.chat integration for voxpupuli/puppet-webhook.
puppet module install puppetlabs/dropsonde
Build some tooling that uses the public data.

Seriously, install the module and classify your primary Puppet server. It would help a ton.

Dropsonde: an expendable weather reconnaissance device created by the National Center for Atmospheric Research, designed to be dropped from an aircraft at altitude over water to measure storm conditions as the device falls to the surface.

Installing Dropsonde is super straightforward. Install the puppetlabs-dropsonde Puppet module and classify your Puppet master with the dropsonde class. If you have multiple compile nodes, just classify the primary Puppet server. This will install the tool and configure a weekly cron job to submit the reports.

Photo by https://unsplash.com/@mattbotsford

Resources and Q&A

While we're doing questions.
You can find the tools I'm talking about at these repositories
And I've been writing about the framework on my blog

PAUSED