Solutions Log » sysadmin

Updating rubygems on Ubuntu

Dan — Mon, 28 Feb 2011 17:37:25 +0000

If you have ever tried to update rubygems on a base Ubuntu 10.04 or 10.10 install you will have been greeted with:

# gem update --system
ERROR: While executing gem ... (RuntimeError)
gem update --system is disabled on Debian. RubyGems can be updated using the official Debian repositories by aptitude or apt-get.

You can work around this limitation:

# gem install rubygems-update
# /var/lib/gems/1.8/bin/update_rubygems

You will now be able to update rubygems using the standard set of commands.

Reference:
http://www.tinymission.com/blog/blogengine.web/post/2010/11/03/Updating-to-Rails-30-on-Ubuntu-1004.aspx

rsyslog consumes excessive memory when forwarding logs with compression

Dan — Thu, 13 Jan 2011 22:10:49 +0000

Issue:
rsyslog consumed growing amounts of memory when forwarding over TCP with compression enabled.

*.* @@(z1)hostname.domain.tld

Cause:
The compression flag causes the rsyslog process to allocate, and never release, memory. I have not determined root cause but the likely culprits are zlib or TLS (both involved when compression is enabled in this situation).

Resolution:
Do not use compression when forwarding with rsyslog. With the compression flag dropped, I was unable to reproduce the errant behavior.

*.* @@hostname.domain.tld

References:
http://www.rsyslog.com/doc/rsyslog_conf_actions.html

/etc/hosts entries not being used for non-root users

Dan — Tue, 26 Oct 2010 18:13:17 +0000

Issue:
Non-root users are unable to resolve addresses for entries in /etc/hosts.

user@gentoo ~ $ ping localhost
ping: unknown host localhost

Troubleshooting:

Potential causes:

Malformed content or improper permissions on /etc/hosts
Malformed content or improper permissions on /etc/host.conf
Malformed content or improper permissions on /etc/nsswitch.conf

Diagnostics:
One may consider nslookup or dig to be suitable diagnostic tools given the use case, however, they are applicable only when troubleshooting DNS server name resolution issues; these tools do not bother to look at /etc/hosts. strace is suitable given the diagnostic task at hand.

user@gentoo ~ $ strace -e open ping localhost
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/lib/libc.so.6", O_RDONLY) = 3
open("/etc/resolv.conf", O_RDONLY) = 3
open("/etc/resolv.conf", O_RDONLY) = 3
open("/etc/nsswitch.conf", O_RDONLY) = -1 EACCES (Permission denied)
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/lib/libnss_dns.so.2", O_RDONLY) = 3
open("/lib/libresolv.so.2", O_RDONLY) = 3
open("/etc/host.conf", O_RDONLY) = 3
ping: unknown host localhost

Cause:
As reported by our strace diagnostic, our user has insufficient privileges to read /etc/nsswitch.conf resulting in the file never being utilized for name resolution. In our case, /etc/nsswitch.conf had a permission mode of 600, allowing only the owner (root in our case) read/write access.

Steps to Reproduce:
Change the mode of /etc/resolv.conf, /etc/host.conf, /etc/hosts to 600.

Resolution:
Change the mode of /etc/nsswitch.conf to 644.

user@gentoo ~ $ sudo chomod 644 /etc/nsswitch.conf
user@gentoo ~ $ strace -e open ping localhost
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/lib/libc.so.6", O_RDONLY) = 3
open("/etc/resolv.conf", O_RDONLY) = 3
open("/etc/resolv.conf", O_RDONLY) = 3
open("/etc/nsswitch.conf", O_RDONLY) = 3
open("/etc/ld.so.cache", O_RDONLY) = 3
open("/lib/libnss_files.so.2", O_RDONLY) = 3
open("/etc/host.conf", O_RDONLY) = 3
open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
ping: icmp open socket: Operation not permitted
user@gentoo ~ $ ping -c 2 localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_req=1 ttl=64 time=0.066 ms
64 bytes from localhost (127.0.0.1): icmp_req=2 ttl=64 time=0.056 ms
--- localhost ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.056/0.061/0.066/0.005 ms

Note: We expect "ping: icmp open socket: Operation not permitted" -- non-root users are not allowed to open raw sockets.

The device, \Device\Ide\iaStor0, did not respond within the timeout period

Dan — Thu, 25 Mar 2010 16:38:21 +0000

Issue:
After an updated release of Intel's Matrix Storage Manager [v.8.9.0.1023] and chipset drivers for Windows, you experience unexpected system timeouts, lockups, pausing, or freezing. The following event is recorded in the System event log:

Log Name: System
Source: iaStor
EventID: 9
Level: Error
The device, \Device\Ide\iaStor0, did not respond within the timeout period.

Cause:
The issue is the result of Aggressive Link State Power Management (ALPM) on the PCI-Express bus negotiating a lower power state for the link between the controller and disk when there is no activity. When ALPM works, disk requests are queued, the serial link revived, and the queued requests are sent to the relevant disk; this requires a disk that supports ALPM.

Resolution:
Modify the advanced settings of your active power management scheme in Windows to turn PCI Express Link State Power Management off.

Commentary:
Searching for a solution yielded a number of possibilities.

Setting the value of:

HKLM\SYSTEM\CurrentControlSet\Services\iaSTOR\Parameters\PortN\LPMDSTATE 0

as discussed by Derek Seaman did not resolve my issue.

Renaming:

HKLM\SYSTEM\CurrentControlSet\Services\iaSTOR\Parameters

HKLM\SYSTEM\CurrentControlSet\Services\iaSTOR\Parameters.dist

as suggested by Paul's Computer Service was also ineffective. At the time I was running the latest set of drivers for my platform. Caveat emptor: the solution I outlined in this article worked for me.

References:
Intel CS-025783 - Possible issues with Windows Vista* and Intel® RAID
Windows 7 Intel SATA/AHCI Lockups
Intel® Matrix Storage Manager Bug

Reformat and reindent an XML file

Dan — Thu, 11 Feb 2010 18:14:53 +0000

This is easy to accomplish with xmllint and a shell one-liner:

xmllint --format inputfile.xml > outputfile.xml

xinetd per_source limit issues

Dan — Tue, 09 Feb 2010 09:32:34 +0000

Issue:
Users note availability issues when accessing services backed by xinetd (subversion, rsync, etc.)

Identification:
Syslog on the affected server will present multiple lines containing daemon per_source_limit for 0.0.0.0.

Cause:
You have exceeded per_source_limit defaults imposed by your xinetd configuration. Many distributions include per_source limits that may not be suitable for your use case. Evaluate your needs carefully.

Resolution:
Modify the default setting for per_source in /etc/xinetd.conf or modify the service specific configuration (recommended) under /etc/xinet.d. per_source limits may be set as follows:

per_source = 10

per_source may be set to an integer or UNLIMITED (the number represents the number of connections allowed per host). A sensible fixed value is always better than UNLIMITED.

Reference: xinetd.conf(5)

Reset HP Integrated Lights Out Board to Factory Defaults

Dan — Tue, 05 Jan 2010 21:45:19 +0000

Issue:
Cannot login to HP Integrated Lights Out Board (bad credentials).

Cause:
Someone failed to record the configuration.

Solution:
If an operating system supported by HP Proliant Essentials is installed along with a recent Proliant Support Pack, you may reset the Integrated Lights Out board to its factory defaults. Use the HP Lights-Out Online Configuration utility.

C:\Program Files\HP\hponcfg\hponcfg.exe /reset

Honor Guest IDE/SATA Flush Requests

Dan — Mon, 07 Dec 2009 20:43:59 +0000

Issue:
VirtualBox ignores IDE/SATA disk flush requests.

Cause:
This is by design. To improve performance VirtualBox ignores these requests. This can create an issue with filesystems that do not have fsck-like tools (ZFS is an example) create a situation where the on disk information is inconsistent due to data held in RAM.

Solution:
To enable flushing for IDE disks, issue the following command:

VBoxManage setextradata VMNAME
"VBoxInternal/Devices/piix3ide/0/LUN#[x]/Config/IgnoreFlush" 0

The value [x] that selects the disk is 0 for the master device on the first channel, 1 for the slave device on the first channel, 2 for the master device on the second channel or 3 for the master device on the second channel.

To enable flushing for SATA disks, issue the following command:

VBoxManage setextradata VMNAME
"VBoxInternal/Devices/ahci/0/LUN#[x]/Config/IgnoreFlush" 0

The value [x] that selects the disk can be a value between 0 and 29.

Reference: http://www.virtualbox.org/manual/UserManual.html#id2499944

Replace Pipes with Tabs in a Delimited File

Dan — Fri, 04 Dec 2009 20:44:37 +0000

Issue:
It is often necessary to replace delimiters in a file with a form the receiving party expects. sed is my favorite method of meeting the need.

Solution:
Replace double pipes with tabs

sed 's/||/ /g' file.in > file.out

Note: If you find the tab key simply does not work, try CTRL+V+I from your terminal.
Reference: http://forums.devshed.com/unix-help-35/replacing-tabs-with-spaces-372623.html

Caveats:
Special consideration (and a regex) will be required for data where delimiters are present in the data itself. Consider your use case and apply appropriately.

Remove a Range of Unwanted ZFS Snapshots from the Command Line

Dan — Thu, 26 Nov 2009 05:51:41 +0000

Remove a range of unwanted snapshots, from the command line, if necessary.
In the following example, all automatic snapshots in the bash shell are removed.

for s in $(zfs list -H -o name -t snapshot | grep @zfs-auto-snap); do zfs destroy $s; done

One could add a secondary pipe and grep statement to filter on individual file systems.

Reference: How to Manage the Automatic ZFS Snapshot Service