So, I have a Wavefront proxy running on Solaris: now I want to get metrics into it.
For a long time,
collectd has been the standard way of collecting
telemetry and pushing to an endpoint, usually Graphite. Wavefront can
intepret the Graphite wire format, so that should the be path of least
Oracle do not provide us with a conveniently installable Collectd
package. But given the dependency chains which can result from
various plugins, this may not be a bad thing. (If you want a giggle
about how someone somewhere thinks Unix systems ought to be built,
have a look at the dependency chain for the Ubuntu
Collectd 5.5 builds easily enough on Solaris though, and I soon had it up and running, sending data to my Wavefront proxy in Graphite format. You’ll get all the basics: CPU, memory usage, disk consumption and throughput, network packets sent and recieved. It works perfectly well.
Collectd feels, I think, a bit old, clunky, and arcane. It has real
problems reconnecting when its endpoint goes away. And, it doesn’t
have stellar Solaris support. Some of the collectors simply refuse to
work, others, like CPU, send limited, whilst the old
plugin haemorrages crap when it sees a box with a couple of hundred ZFS
With this in mind, and fancying learning something new, I decided to have a go with Diamond. This is a more modern approach to the same problem. It’s written in Python, and it’s very modular, so it ought to be easy to extend. But, it’s not without problems.
Problem 1: Installing Diamond
Installing Diamond on Solaris is easy:
# pip install diamond
Except it isn’t if you don’t have a C compiler. Diamond requires
psutil (even though it doesn’t actually work properly: more later)
psutil module needs to compile itself. That’s fine for a
dev box, but even in the age of the 2Gb base install, I am not
putting GCC or Forte (God, I’m old) on machines just to install a
single Python module
Problem 2: Nothing Works
Trying out the standard collectors didn’t go too well. In short:
CPUCollector: this works, but it’s pretty basic, just giving you the user, system, and idle times for each VCPU. So nothing on interrupts, cross-calls, context-switches or any of that good
MemoryCollector: doesn’t work at all, and doesn’t even fail gracefully. For some reason the
psutilimplementation seems incomplete on Solaris. It probably uses
NetworkCollector: this doesn’t work, for the same reasons, and with the same bad error-handling, as
LoadAverageCollector: this works, but who cares? Load average is rubbish.
DiskUsageCollector: doesn’t work because it uses
/proc/diskstats, which we obviously don’t have.
DiskSpaceCollector: doesn’t work in the global zone when running as a non-root user with standard privileges, because delegated datasets are not visible from the global. But, I don’t really care about filesystem usage, as everything is in ZFS datasets. For the most part, I’m more interested in the capacity of the pools, and this can’t do that.
VMStatCollector: doesn’t work because, guess what? It uses
/proc. (And even if it didn’t, Linux’s
vmstatis different to the one in Solaris.)
NFSCollector: doesn’t work. You know why.
Well, I’m up slack alley with that lot aren’t I? And there’s nothing for ZFS, or zones, or SMF, or FMA, or pretty much anything that I might conceivably want to alert on.
Problem 3: Virtualization
One of the great things about zones is that a non-global-zone (NGZ), is fully visible from the global. This gives you a different way to perform many housekeeping jobs. Why run dozens, or hundreds of identical jobs in all the zones on a box when you could run the same job once from the global?
As you think more deeply about telemetry on a zoned system, you find yourself dividing metrics down into particular scopes.
For instance, all zones see the same physical disks, so monitoring disk I/O across a hundred zones gives you a hundred copies of the same metric. But counting failed services needs to be done per-zone. You can’t run an NFS server from an NGZ, but you can see the kstats for one running in the global.
Things get more complicated in a multi-tenant system: what metrics do your users care about? Will global things visible from their NGZ confuse them?
If you’re running the system, you can see how much of their allocated disk space your clients are using, but why do you care – it’s no skin off your nose if they fill their quota. How about VNIC throughput? That might be relevant to you and your tenants.. As so often we find that the technological problem of recording this stuff is a small part of the whole.
I know a little bit of Python, and I know a bit about Solaris internals and the user interfaces to them. Get writing!
Solution 1: Installing Diamond – use
To get round Python dependencies, need for a C compiler etc, I took the “Omnibus” approach that I use to deploy Puppet.
That is, I compiled Python 2.7 from source, installed it in
/opt/diamond, then installed the PIP in there. I packaged up that
directory (actually I
tarred it: that’s still a package, ask
Slackware) and now I can drop it into any box or zone, however minimal.
This also has the nice side-effect of meaning that changes to the system
Python can’t break your Diamond install.
If I were doing this on SmartOS I’d make a proper
pkgin package: it’s
not a lot more effort than a tarfile. IPS, though, is way too much of a
Solution 2: Nothing Works – use
Solaris does have a
/proc filesystem, but it is quite a pure
interface, only exposing
process information, rather than Linux’s
“chuck anything in there, in any format, possibly even binary, and
oh, you can write to some of it” approach.
Instead, we have kstats. Kstats are kernel counters, exposed to userland
/dev/kstat, which appears in any kind of zone. Solaris ships with C
and Perl libraries which provide simple access to kstats, and bindings
exist for a number of other languages. Fortunately for me, someone
already made a Python binding, and
someone else made it better.
Kstats are good and bad. First, the good. They are quick to access. They are very accurate: values are given with a high-resolution timestamp. They give you deep introspection – look:
$ kstat | wc -l 67150
That’s a lot of information for one host. I wouldn’t want the bill for chucking a datacentre full of those into Wavefront every second.
The bad, now. kstats aren’t documented. This is because they are not considered a stable interface. They can change. They can go away, be renamed. and new ones can appear. (It’s actually not hard to add your own.)
This lack of documentation also makes it difficult to work out what
a lot of them are. Off the top of my head I couldn’t tell you the
exact meaning of
sdpib:0:sdpstat:sdpOutUrg, and finding out might
mean a trip deep into the
source. (Not the Solaris
source, obviously. Up yours, Oracle.)
The final difficulty with kstats is that, in statsd speak, some are counters, and some are gauges. For instance, the number of pages in the free list is a gauge: it goes up and down.
$ while true; do; kstat -p unix:0:system_pages:pagesfree; sleep 1; done unix:0:system_pages:pagesfree 234875 unix:0:system_pages:pagesfree 235084 unix:0:system_pages:pagesfree 234786 unix:0:system_pages:pagesfree 234924 unix:0:system_pages:pagesfree 234905
Whilst the bytes written to a disk is a counter which only ever goes up:
$ while true; do; kstat -p cmdk:2:cmdk2:writes; sleep 1; done cmdk:2:cmdk2:writes 1488501 cmdk:2:cmdk2:writes 1488509 cmdk:2:cmdk2:writes 1488518 cmdk:2:cmdk2:writes 1488526
cpu_info:0:cpu_info0:chip_id are constants, and
cmdkerror:0:cmdk0,error:Model are constant strings. All
of this is fine, and makes good sense, but it’s not possible to
programatically tell which kstat is of which datatype, and sometimes
difficult to work it out from man pages and source code.
I also, at the time of writing, am ambivalent about the correct way to
deal with the counter kstats. When you get a group of kstats, that
snaptime, which is a high-resolution timestamp
telling you exactly when the kstats had the values you see. If you
have Diamond remember the last value of
snaptime, it’s therefore
trivial to calculate and send deltas rather than absolute values.
Because, that’s what we want, right? Hmmm, weeeeelll, it could be.
Take traffic through a NIC. My immediate reaction is to think rate is
what matters. But, if I want to see which VNIC has used the most
traffic, or get an absolute value for usage over a 24h period, I have to
do some work summing all those little deltas. (Though the Wavefront
CLI, particularly with
human formatter, makes this relatively simple, especially if
you know a little bit of
awk. And if you don’t know a little bit
awk, learn a little bit of
If I send the counter, these tasks become trivial, and displaying
the rate on a graph is as simple as wrapping the timeseries in
deriv() function. Or, maybe I’ll decide that looking
at the steepness of the unmodified counter line gives me a better
sense of how all my NICs are doing. I think the raw value gives you
flexibility, which is good. But, hasn’t Collectd set the precedent?
Don’t people expect to get rates? It’s hard to decide what’s right.
Using the kstat
snaptime lets you calculate very, very accurate
rates, but does that really matter? Most people are used to using
Graphite, with 15-second intervals, and everything most likely
rounded to an integer. And,
snaptime might not be what you think
it is. Are you thinking it’s the clock time? Seconds since the
epoch? I bet you are, but you’re wrong. It’s the time elapsed since
a point marked by its friend
crtime. This is also a
high-resolution timestamp, and is time the kstat was created.
Relative to… errr…. yes.
Time is expressed as nanoseconds since some arbitrary time in the past; it is not correlated in any way to the time of day
so, good luck converting your kstat
snaptimes to anything meaningful.
It’s not ideal for our particular application, but I can see why
things were done this way. Calulating a rate since
you a (probably pointless) “since boot” type summary, like the first
vmstat output. Clearly kstats and their
implemented with things like
mpstat in mind: accurate
“live” deltas, not pinned to a fixed point in time. But feeding data
into a telemetry system mandates that you fix it to timestamps.
We’re at odds, aren’t we?
New Shiny! Point Tags
I particularly like Wavefront’s ability to apply any number of
key=value tags to a point. This lets you build up
multi-dimensional data. Tags are fully integrated in the query
language, so you can use them for all manner of selection and
filtering. They’re also handy markers, giving clues that might be
useful in further investigation.
At my current client’s site, we have a Puppet reporter which, at the end of each run, pumps all the statistics for that run (times, change and error counts etc.) into Wavefront. It tags all these points with the release of the Puppet code, so if we see, say, a spike in run time, or something starts making flappy changes, it’s easy to know which Puppet release caused the problem. We can also easily find any machines which aren’t on the same release but should be.
Starting simple, one of the first Solaris collectors I wrote was to
A lot of people don’t like Solaris’s SYSV
c0t0d0s0 way of
identifying disks, but kstats don’t even do that: they use the
kernel module to derive the disk names, and call them things like
cmdk0 (IDE or SATA, obviously), or the somewhat more logical
for SCSI or SAS. These names are not particularly meaningful, and I
wanted a better way of knowing which disk was on the blink when my
disk_health alerts fired.
In among the
sderr kstat bundle is the serial number
of the disk. “Wouldn’t it be nice”, I thought (knowing full well
most tech disasters begin with just those words), to tag the points
so you can more easily idenfity the errant disk? Which brings us to:
Problem 4: Diamond does not Understand Point Tags
Up to this point, I was using the OpenTSDB handler (output plugin) for Diamond. OpenTSDB does have a concept of tags, but they’re different to point tags, so it was time to…
Solution 4: Write a Wavefront Handler
Wavefront’s wire format is, of course, clearly documented, and very close to OpenTSDB’s, so I began with the code for the OpenTSDB handler, and soon ended up with a working Wavefront equivalent. There’s not a lot to it: with all of these things you only ever end up poking a simple string into a socket.
Next there were some pretty minor changes to the Diamond metrics code to give us an extra argument for defining point tags. Diamond’s code is simple, and well structured, so making changes isn’t hard at all.
Now, I was able to return to my collector, pull each drive’s serial number out of the kstats, and apply it to every error point for the relevant disk. This makes it far easier to identify the exact disk which is reporting errors. I also tag (when I can: not all devices expose the same information) by vendor and model, because in large-scale deployments it might be interesting to see whose devices show the highest error rates. Or maybe the highest throughput. Or anything else you can think of.
Now, when you hover over a point on a disk error chart, you see:
It may not be the most useful thing in the world, but I hope that is reasonably illustrative of point tags. I think they are a powerful feature of Wavefront, and I expect to get a lot of value from them in the future. At the moment, it’s a little hard to know how best to get data tagged, and I hope the code I’ve written is a step along the road.
My Diamond extensions allow you to tag per-point, in the collector code, or en-masse, tagging everything from a single collector, or even everything from the whole of Diamond.
I’d like to go even further, and be able to apply point tags en-masse in the proxy, perhaps using regexes, but that’s one for Wavefront.
At the time of writing, all of this work is sitting in a PR in Diamond’s repository. I don’t know when, or if, it will be merged into master, but if you wish to use it now, you can clone from my branch, and if you can make it better, piggy-back off my PR.
Postscript: There is no Solution 3
The observant reader (yeah, right) may have noticed that I posed four problems but only three solutions; the outstanding being “from where to collect what” on a heavily virtualized system.
I’m still mulling it over. Still changing my mind, and still experimenting with different approaches.
Most of the new collectors I have written so far are for a full-machine overview: that is, they are intended to run in the global zone. Others adapt their behaviour depending on their view of the system. This seems to be working pretty well, but the requirements for a multi-tenant system are different. As always with open source software, I’m scratching my own itch, and that itch is mostly a server where all the zones are mine.
I do have a couple of instances in the Joyent Public Cloud, for which I’ve written a collector solely for use in a SmartOS zone. That collector is to help keep check on how close you are sailing to the limits of the instance. It is instrumenting the site you are looking at now.
Solaris’s introspection goes very deep: it’s always been a very
observable platform, and it’s a lot of fun feeding all those numbers
into Wavefront. I’ve always been interested in system interals, and
it’s quite exciting to be able to properly visualize things like
latency, saturation, paging, batching of writes and anything else
that I can think of. I like to stick a bunch of different metrics on
a single chart, wrap them all with the
normalize() function, and
see how work is distributed throughout the system: see the network
throughput spike, then the CPUs start processing data, and handing
the results to disk; and I can watch how the SSD ZFS cache device
interacts with, and behaves differently to the spinning disk where
the data ends up.
It’s informative and fun to generate more and more load on the box and watch how things fail. Or, rather, how things behave differently in order not to fail. That, I think, would make for a good article in the future.
Next time, though, I’m going to have a crack at wiring DTrace up to Wavefront. If you think there are too many kstats to choose from, wait until you see what DTrace offers.