Solutions Log » Shell

Reformat and reindent an XML file

Dan — Thu, 11 Feb 2010 18:14:53 +0000

This is easy to accomplish with xmllint and a shell one-liner:

xmllint --format inputfile.xml > outputfile.xml

Replace Pipes with Tabs in a Delimited File

Dan — Fri, 04 Dec 2009 20:44:37 +0000

Issue:
It is often necessary to replace delimiters in a file with a form the receiving party expects. sed is my favorite method of meeting the need.

Solution:
Replace double pipes with tabs

sed 's/||/ /g' file.in > file.out

Note: If you find the tab key simply does not work, try CTRL+V+I from your terminal.
Reference: http://forums.devshed.com/unix-help-35/replacing-tabs-with-spaces-372623.html

Caveats:
Special consideration (and a regex) will be required for data where delimiters are present in the data itself. Consider your use case and apply appropriately.

Create a pidof command to find PID numbers easily

Dan — Wed, 07 Oct 2009 20:30:44 +0000

Most UNIX environments include the pidof command which is put to use whenever one needs to quickly determine, by name, the pid of a running program. Apple's Mac OS X lacks the pidof command, however, one may approximate its function with the following shell script:

#!/bin/sh
ps axc|awk "{if (\$5==\"$1\") print \$1}";

Save the script as /bin/pidof and be sure to set its executable bit:

chmod a+x /bin/pidof

Strip empty (null) lines from a file

Dan — Tue, 06 Oct 2009 18:12:06 +0000

sed meets the need; the recipe follows:

sed '/^$/d' filename

rsync failed to set times on [filename]

Dan — Fri, 10 Jul 2009 02:34:52 +0000

This error occurs because the version of rsync on the system cannot preserve modified times for directories.

Run rsync with the following arguments to suppress this warning:

 rsync -avPO ./source/* ./destination

Explanation of switches:

1
2
3
4

a -> Archive mode (do not preserve hard links, ACLs, or extended attributes)
v -> Verbose (I like to know what is happening)
P -> Equivalent to --partial --progress (for long transfers that may be interrupted)
O -> Omit directories from times

Create thumbnails en-masse from a bash prompt

Dan — Wed, 08 Jul 2009 23:10:14 +0000

A simple one-liner and ffmpeg, basename, and cut gets this done.

1	for i in *.f4v; do ffmpeg -i `basename $i` -s 320x240 `basename $i \| cut -d'.' -f1`.jpg; done

Thumbnail output size is configurable with the -s switch.

Count the number of characters (or columns) in a file

Dan — Thu, 25 Jun 2009 20:58:41 +0000

Sometimes you want to count the number of characters or columns in a file.

1	awk 'length>max{max=length}END{print max}' filename.txt

For extraordinarily long or wide data sets you may want to consider the following:

1	head -n 10 \| awk 'length>max{max=length}END{print max}'

1	tail -n 10 \| awk 'length>max{max=length}END{print max}'

Convert text file to UTF-8 encoding from the shell

Dan — Wed, 24 Jun 2009 15:38:45 +0000

1	for f in *.txt; do iconv -f mac -t utf-8 "$f" >"$f.utf8"; done

Note:
-f Input file encoding type
-t Output file encoding type

Reference: http://manual.macromates.com/en/saving_files.html

Mac OSX does not contain seq

Dan — Mon, 15 Jun 2009 20:07:15 +0000

seq is used for generating a series of numbers.

Mac OSX does not contain this. To overcome the limitation, install MacPorts and issue the following command:

1	sudo port install seq

You will also note that seq is not available from a shell prompt, that is because it exists as gseq. Symbolically link it:

1	sudo ln -s /opt/local/bin/gseq /opt/local/bin/seq

Sort files according to their size

Dan — Thu, 11 Jun 2009 20:07:45 +0000

1	du -s * \| sort -n