— modern ops stuff —
ipadm (and IPMP on Solaris 11)
30 April 2012 // Solaris

The ipadm command is new in Solaris 11 and the Illumos derived distributions. It’s a different way of handling IP interfaces, replacing ifconfig and a lot of /etc file manipulation.

IP interfaces

Creating an Interface

The first, most basic operation for which to use ipadm is the creation of a new IP interface. You need an existing datalink to create an interface on - note I say datalink, not NIC. That’s a good job, because I don’t have any free physical NICs on this box, so I’ll have to create a virtual NIC on top of atge0 and use that.

# dladm create-vnic -l atge0 test0
# ipadm create-ip test0

The ipadm command may seem like a pointless step. What, really, is an “interface”? It doesn’t have an address or any useful properties, so it doesn’t really do anything. Why can’t you just assign the address, netmask etc. to the link, like you always used to? Well, though I created a (typical) IP interface, you can also create VNI interfaces and IPMP groups with create-vni and create-ipmp respectively. (More of the latter will follow.) So, we had to tell ipadm what class of link we wanted to make. You can see all the interfaces, and their classes, with

# ipadm show-if
IFNAME     CLASS    STATE    ACTIVE OVER
lo0        loopback ok       yes    --
atge0      ip       ok       yes    --
test0      ip       down     no     --

Assigning an Address

So there’s my new interface. It’s not a lot of good though, is it? It’s “down” and not “active”, and it doesn’t have an address. Let’s create a static IP address on the new interface.

# ipadm create-addr -T static -a local=192.168.1.155/24 test0/addr

The create-addr part is fairly obvious - we’re creating an address. -T is the type of address we’re creating, in this case, static (it can also be addrconf if you’re using IPv6, which I’m guessing you probably aren’t, or dhcp if you want to automatically configure your manually configured IP address.) The -a option sets the address, which can be local, as ours is, or remote. (remote is for point-to-point connections, where you have to specify the remote and local endpoints as local=1.2.3.4,remote=15.6.7.8). If you’re just creating a “normal” local interface, you can omit local=. You can leave out the netmask /24 in our example as well, assuming you’ve got suitably configured /etc/netmasks file, and if you have a suitable entry in hosts it’s even possible to use a hostname instead of an IP address.

The final argument, net0/addr is the “address object”. An interface can have multiple address objects associated with it, but often will have just one - the IPv4 address. You can examine addresses with ipadm show-addr, and use the address object name to be specific.

# ipadm show-addr test0/addr
ADDROBJ           TYPE     STATE        ADDR
test0/addr        static   ok           192.168.1.155/24

That looks pretty much configured to me. Let’s quickly check it with ifconfig (I won’t tell Oracle if you don’t.)

$ ifconfig test0
test0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
        inet 192.168.1.155 netmask ffffff00 broadcast 192.168.1.255
	    ether 2:8:20:7f:7d:a4

Note that the interface is UP. This is the default, and if you want it DOWN, throw -d into the create-addr command. If you don’t want the address to persist (i.e. it only lasts until the next reboot) add in -t, like you do with svcadm.

I don’t know about you, but I rather liked that as a way of configuring an interface. It seems very clean and simple. show-addr even notifies you of IP address collisions by putting duplicate in the STATE field! The only oddity is that “address object” we have to keep tagging on the end. That was weird, and pointless, right?

Well, no. For a start, the address object name gives you a way to refer to the address. Say you wanted to delete the IP address, but not go so far as blowing away the VNIC, you’d specify the object:

# ipadm delete-addr test0/addr

There’s nothing magical about the word addr either.

# ipadm create-addr -T static -a 192.168.1.155/24 test0/live

is perfectly valid, and perhaps tells us that interface is on the “live” VLAN. (I’m very big on self-documenting naming schemes.) Or perhaps if you were to have IPv4 and IPv6 on the same interface, you might call them net0/v4 and net0/v6 or something?

Address object names also allow you to give sensible names to virtual IP addresses when using shared IP instances for zones. (Though you probably won’t use shared IP instances in Solaris 11, as the exclusive instance configuration is so good.)

All of the above works in OpenIndiana too. If you’re using Solaris 11, you’re into the hell of configuring your DNS via SMF. If not, lucky you!

Setting a Default Route

To continue the theme of not editing files in /etc we define our default router through a command now as well:

# route -p add default 192.168.1.1

We’ve had route -p for a while now I think, since fairly early on in the Solaris 10 cycle, so that’s probably worked for ages.

Managing and Tuning Interfaces

Want to briefly take an interface down, then bring it back up? Easy.

# ipadm down-addr test0/live
$ ping 192.168.1.155 1
no answer from 192.168.1.155
# ipadm up-addr test0/live
$ ping 192.168.1.155
192.168.1.155 is alive

You can disable and re-enable the interface itself just as efficiently with the disable-if and enable-if subcommands.

Have you ever had to tune your TCP stack? Writing rc.d scripts that fiddled with ndd every reboot never seemed right, did it? Try running

# ipadm show-prop

And there are all the tunables, nicely laid out and easy to read and manipulate. Just want the TCP properties? No problem:

# ipadm show-prop tcp

It even shows you allowable ranges and values! Want to increase your receive buffer size until next reboot?

# ipadm set-prop -t -p recv_buf=300000 tcp

This gets better all the time doesn’t it? To make the change persist across reboot (i.e. not transient), omit -t.

We’ve just been looking at interface properties. Address objects have properties too, and

# ipadm show-addrprop

will show you them for all objects. To query a single object, out comes the object name again:

# ipadm show-addrprop test0/live

There is a third set of properties which can be viewed and manipulated with ipadm, and they belong to the datalinks, both physical and virtual.

# ipadm show-ifprop

Want to change the IPv4 metric on atge0?

# ipadm set-ifprop -p metric=1 -m ipv4 atge0

What, that’s broken all your networking and want to change it back to the default?

# ipadm reset-ifprop -p metric -m ipv4 atge0

Hopefully by now you’ll have realized that ipadm is a Good Thing, pulling together stuff that was previously scattered all over Solaris, and putting a good clean front end on it. It even produces colon-separated machine-parseable output with the -c and -o options. I don’t know if it’s the same people who did dladm, but it’s an equally good bit of design and implementation.

Getting Rid of an Address

You can delete an interface even if it has live addresses.

# ipadm show-addr  test0/addr
ADDROBJ           TYPE     STATE        ADDR
test0/addr        static   ok           192.168.1.155/24
# ipadm delete-ip test0
# ipadm show-addr  test0/addr
ipadm: address object not found

IPMP

I said earlier that an interface can be IP, VNI, or IPMP. I’ve never used a VNI link, so I don’t feel qualified to talk about that, but I do use IPMP, and I always felt the way you managed it in previous versions of Solaris was something of a mess. Here’s where we genuinely do need a new command, and that command is, of course, ipadm.

For the sake of illustration I’m going to create an IPMP pair using two VNICs on the same physical interface. Obviously that’s pointless and stupid in real life, but it’ll do here.

# dladm create-vnic -l atge0 path0 dladm create-vnic -l atge0 path1

You need an IP interface on both of those paths:

# ipadm create-ip path0
# ipadm create-ip path1

Then you can create your IPMP group using those interfaces.

# ipadm create-ipmp -i path0,path1 ipmp0
# ipadm show-if ipmp0
IFNAME     CLASS    STATE    ACTIVE OVER
ipmp0      ipmp     down     no     path0 path1

Now you can treat ipmp0 (you could have called it anything you wanted) as if it were any other interface. So let’s give it an address:

# ipadm create-addr -T static -a 192.168.1.177/24 ipmp0/v4
$ ping 192.168.1.177
192.168.1.177 is alive

You can view and change the properties of that address just as if it were a “normal” SPOF interface. And, of course, that IPMP group will still be there if we reboot. Ridiculously simple isn’t it? You know what, I don’t think I trust that two-path group. Let’s add another.

# dladm create-vnic -l atge0 path2
# ipadm create-ip path2
# ipadm add-ipmp -i path2 ipmp0
# ipadm show-if ipmp0
IFNAME     CLASS    STATE    ACTIVE OVER
ipmp0      ipmp     ok       yes    path0 path1 path2

On second thoughts, maybe that’s overkill. To remove “the third way”:

# ipadm remove-ipmp -i path2 ipmp0

To digress slightly, there’s also a command called ipmpstat which can give you more information on things like the health of the group (with -g), the frequency and routing of probes (-p), and the test addresses those probes use.

# ipmpstat -t
INTERFACE   MODE       TESTADDR            TARGETS
path1       disabled   --                  --
path0       disabled   --                  --

Clearly, the group we set up doesn’t use test addresses, so it’s not the safest form of IPMP. We can add test addresses like so:

# ipadm create-addr -T static -a 192.168.1.181 path1/testv4
# ipadm create-addr -T static -a 192.168.1.180 path0/testv4
# ipmpstat -nt
INTERFACE   MODE       TESTADDR            TARGETS
path1       routes     192.168.1.181       192.168.1.1
path0       routes     192.168.1.180       192.168.1.1

Note how I chose to use a descriptive address object name, which I think is good practice. The -n flag in the ipmpstat command forces addresses to be printed as numbers. Otherwise lookups are done and hostnames are printed. I thought that output was less clear for my example.

Removing IPMP interfaces

You can’t just blow away an IPMP interface like you can a normal IP interface

# ipadm delete-ipmp ipmp0
ipadm: cannot delete IPMP interface ipmp0: IPMP group is not empty

We need to remove the interfaces from the IPMP group:

# ipmpstat -t
INTERFACE   MODE       TESTADDR            TARGETS
path2       disabled   --                  --
path1       routes     192.168.1.181       router
# ipadm remove-ipmp -i path1,path2 ipmp0
# ipadm delete-ipmp ipmp0

And for the sake of tidiness, remove those interfaces too.

# ipadm delete-ip path1
# ipadm delete-ip path2

So that’s the new way of doing network interfaces and IPMP. Good, isn’t it?

Tags: