Initialize `/etc/hosts` in zones

[rmustacc]

The other day @jgallagher got the following messages on the console:

14Jul  9 03:10:00 oxz_nexus cron[1796]: [ID 574227 user.alert] Solaris_audit getaddrinfo(oxz_nexus) failed[node name or service name not known]: Error 0
Jul  9 03:10:00 oxz_nexus cron[1796]: [ID 574227 user.alert] Solaris_audit mapping oxz_nexus to fe80::8:20ff:fe97:6ad0: Error 0

These come from the auditing subsystem in illumos. In particular, the reason that these console logs are generated is due to the fact that we're trying to resolve the hosts primary node name to a DNS address, but failing to. While we could opt to wait until we have DNS set up, it might make sense to make a mapping of the node name as an entry in /etc/hosts. In fact, the default /etc/hosts that we have right now just has mappings for local host. So if we look at say the file in the nexus zone (oxc_nexus), we just see (stripping the comments out):

::1             localhost
127.0.0.1       localhost loghost

On other hosts you might actually see the system's name in /etc/hosts when it's not part of the DNS scheme. From here we have two options that we may want to consider:

1. Have sled-agent append an entry for the IPv6 address as the node name as a line in /etc/hosts as part of all of its other static networking setup.
2. Assume that DNS will always be functioning and allow that to work.

I have a minor, perhaps irrational preference, for (1). In particular, the main idea here is that if DNS isn't functioning or not to rely on the dns search path or related features, it may make sense to still have a fixed entry. In particular, because it mimics what we're trying to do and will help speed up the overall process when using getaddrinfo() for processing by the system. Note, that the DNS bits would still be needed regardless for the rest of the system to reach it and for the individual services which will be using a cueball like system that'll always use DNS and therefore bypass the name service switch and related.

@jclulow thoughts on this part of the network naming?

[smklein]

I have a couple clarifying questions:

The oxz_nexus zone has several addresses - here's one example:

ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
oxControlPublic0/public static ok       192.168.1.20/24
lo0/v6            static   ok           ::1/128
oxControlService2/linklocal addrconf ok fe80::8:20ff:fec5:f7cf/10
oxControlService2/omicron6 static ok    fd00:1122:3344:101::3/64

When we discuss adding an entry for the system's IPv6 address, we are referring to the 'Internal IPv6 address', used for the internal HTTP interface, which is allocated as omicron6 here, correct?

Additionally, regarding your proposed solution (1), we would want to add entries for all zones on the system, not just Nexus, no? So each time we add / remove a Propolis instance, we'd need to modify /etc/hosts with the new internal addresses, correct?

[rmustacc]

When we discuss adding an entry for the system's IPv6 address, we are referring to the 'Internal IPv6 address', used for the internal HTTP interface, which is allocated as omicron6 here, correct?

So this is a good question in general. So, this is where the node name and host name distinction comes in. We have a node name that we've named this thing (oxc_nexus in this example or replace with a propolis, crucible, etc. zone). It may have multiple host names in DNS e.g. something on the internal network that points to fd00:1122:3344:101::3. And something in external DNS that points to 192.168.1.20. Note, that when we have a future OPTE based world, the oxControlPublic0 will be pretty different in so far as it won't actually have an external IP on it.

In general, I ask myself what would I consider the "primary" which is not a standard technical term in any sense. But generally what do I expect if I walk up to the zone and ask for its IP address for its name. In this case, I generally will expect the omicron6 address as you highlighted and is what I'd use for most things as it represents the majority of the traffic and how we think of the system internally.

Additionally, regarding your proposed solution (1), we would want to add entries for all zones on the system, not just Nexus, no? So each time we add / remove a Propolis instance, we'd need to modify /etc/hosts with the new internal addresses, correct?

Yes, ideally we'd want this in all zones. I used omicron as a reference (since that's what @jgallagher had seen), it was not meant to be limited there.

[jclulow]

Yes I think in general it would probably be good to stick the nodename on the */omicron6 static address in hosts(5) when we set up the IPs -- whether in a global or non-global zone.

[rcgoodfellow]

What functional purpose does making the zone's own hostname locally resolvable serve?

[jclulow]

What functional purpose does making the zone's own hostname locally resolvable serve?

There is a general expectation that the machine's nodename (see uname(2) and utsname.h(3HEAD)) will resolve, through the hosts(5) database, to a valid IP address for the machine. In some cases it's possible for it to be the "primary" IP address of the machine, though not all multi-homed configurations support that.

I believe various bits of software, like the Java standard library, and apparently the BSM/audit stuff, have at times depended on this working. It's true that the nodename is distinct from the (optional) host name that each IP address present on a host might map back to, and that there is nothing that technically enforces that they be aligned, it avoids problems when they do align on some level because of the history here.

[jclulow]

For more context, I believe this error is coming from adt_get_hostIP() which attempts to translate a name to a best IP address for inclusion in audit records. This routine is called by both adt_load_ttyname() and adt_load_hostname(), both of which are often just passing in the value returned by sysinfo(2) with the SI_HOSTNAME command, which is equivalent to the nodename field of struct utsname. These routines are then called by various audit record generation routines to populate "terminal" portion of audit records, which includes an IP address for the machine in question.

In the event that getaddrinfo(3SOCKET) returns that there are no names for the given address, as it is doing here (EAI_NONAME results in the node name or service name not known message we see), the adt_get_hostIP() routine presently falls back to various other options. This includes just picking any IP interface address it can find, or the localhost address if it cannot. I suspect we could actually just delete this message without it being a huge problem.

We still need to populate hosts(5) as described in this issue, but it would at least make this particular message go away.

[jclulow]

I have filed 16040 auditing could be calmer about translation of nodename to IP address for the message.