Lookupd Problems in Apple's Mac OS X Server

Lookupd Problems in Apple's Mac OS X Server
By: Colin Allen and Edward N. Zalta
The Stanford Encyclopedia of Philosophy
May 20, 1999 - 01:27 AM

Reports have surfaced on the internet about Apple's Mac OS X Server's network functions "freezing" for extended periods. Out of the box, OS X Server has all network services turned off and is not vulnerable, but once a system administrator has enabled network services the problem may manifest itself when performing routine network tasks. The problem is reproducible and can result in denial of services for periods up to 35 minutes and beyond. Furthermore, with knowledge of the mechanism, a denial of service attack could be mounted from the internet against machines running Mac OS X Server and OpenStep in some common configurations.

This behavior is related to a bug inherited from NeXT Software Inc's OpenStep. The problem lies in Apple's lookupd, the daemon process through which all programs on the system request information about users and other machines on the network. Lookupd can handle multiple threads, but if given a certain kind of task to do, can become clogged and stop responding to other lookup requests, including password and host authentication for users. Of particular note is that if this happens and the root user does not happen to have an open terminal on the machine then the only options are to wait until the system clears itself or to do a hard reset.

In mid-April we documented the bug in lookupd and sent the results to Apple. Apple is working on the problem, and they plan to release a bug fix before the next release of the OS.

The specific problem involves the domain name system (DNS). Certain IP addresses on the network can cause gethostbyaddr, which on OpenStep and OS X Server calls lookupd, to wait 75 seconds before timing out. If lookupd is hit with enough requests involving such an IP address or addresses, then it becomes blocked and quits responding to all other requests for information. This means that password verification for users attempting to log on is blocked, as well as many other system services. The result is a system that is effectively frozen for as long as it takes for all the domain name lookups to time out. We have seen delays as long as 35 minutes with less than 20 such requests.

Systems may be vulnerable to the problem from internet clients using various protocols. For instance, with the Apache web server configured to do hostname lookups, the lookupd blockage can be caused by the receipt of repeated http requests from any machine on the net that is configured with an address that causes the domain name lookup to time out. Multiple requests can be generated innocently by web surfers clicking multiple times on a link when the server fails to respond, or by an unlucky confluence of users with problematic addresses hitting the server. By examining our http access logs we have identified dozens of such addresses and by extrapolation to other machines on the same subnet we believe that this is just the tip of an iceberg of thousands of such addresses. Apple ships Apache with HostnameLookups Off but many server administrators turn it on in order to keep better server records.

Even with Apache configured not to do hostname lookups, or not running Apache at all, it is possible to cause Mac OS X Server to freeze. Mac OS X Server systems may be vulnerable to network attack though other services that call lookupd, such as telnet, ssh, and sendmail. However, even if you turn off all internet services, freezes can be caused by any local user with a knowledge of a programming language, process forking, and gethostbyaddr. We have written a Perl script that can be run from any user account and that will cause the system to freeze. The script works simply by forking a number of children that call gethostbyaddr using one of the problematic addresses.

In addition to making sure that Apache is not doing hostname lookups, administrators of Apple Mac OS X Server systems can make two configuration changes in lookupd to reduce the likelihood of a freeze (see Workarounds). The first is to use netinfo to add a NILAgent to lookupd that will cache bad addresses. The second is to increase the number of threads available to lookupd. These workarounds do not eliminate the problem completely.

We have tested our script extensively on linux and have shown that it is not vulnerable to this problem. With Apple's open sourced Darwin code available, developers interested in the lookup daemon should be able to suggest ways to fix this problem. Two avenues for investigation are to prevent multiple lookups of the same IP address, and to reserve some threads for non DNS lookups.

Workaround

You can configure lookupd to make your OS X Server system less vulnerable to the problem

Create a NILAgent to cache problematic addresses (courtesy of Marc Majka @ Apple)

  niutil -create . /locations/lookupd/hosts
  niutil -createprop . /locations/lookupd/hosts LookupOrder CacheAgent NIAgent DNSAgent NILAgent
  niutil -create . /locations/lookupd/agents/NILAgent
  niutil -createprop . /locations/lookupd/agents/NILAgent TimeToLive 3600

Increase the number of threads available to lookupd (courtesy Douglas Davidson @ Apple)

  niutil -createprop . /locations/lookupd MaxThreads 256

The machine must be restarted or lookupd restarted for these changes to take effect. You may also want to turn on lookupd logging as follows:

  niutil -createprop . /locations/lookupd LogFile /tmp/lookupd.log

Further information about lookupd can be found at http://til.info.apple.com/techinfo.nsf/artnum/n24902 although at the time of writing this article this Apple document does not describe the problem identified above.

We would like to acknowledge the help of Susanne Riehemann and Uri Nodelman in tracking down this bug.