Showing posts with label Mac OS. Show all posts
Showing posts with label Mac OS. Show all posts

How to Back Up Directories using Shell Script

Related to the task of backing up entire file systems is the user-centric task of taking a snapshot of a specific directory or directory tree. This simple script allows users to easily create a compressed tar archive of a specified directory.

The Code
#!/bin/sh

# archivedir - Creates a compressed archive of the specified directory.

maxarchivedir=10 # size, in blocks, of 'big' directory
compress=gzip # change to your favorite compress app
progname=$(basename $0)

if [ $# -eq 0 ] ; then
echo "Usage: $progname directory" >&2 ;exit 1
fi

if [ ! -d $1 ] ; then
echo "${progname}: can't find directory $1 to archive." >&2; exit 1
fi

if [ "$(basename $1)" != "$1" -o "$1" = "." ] ; then
echo "${progname}: You must specify a subdirectory" >&2
exit 1
fi

if [ ! -w . ] ; then
echo "${progname}: cannot write archive file to current directory." >&2
exit 1
fi

dirsize="$(du -s $1 | awk '{print $1}')"

if [ $dirsize -gt $maxarchivedir ] ; then
echo -n "Warning: directory $1 is $dirsize blocks. Proceed? [n] "
read answer
answer="$(echo $answer | tr '[:upper:]' '[:lower:]' | cut -c1)"
if [ "$answer" != "y" ] ; then
echo "${progname}: archive of directory $1 canceled." >&2
exit 0
fi
fi
archivename="$(echo $1 | sed 's/$/.tgz/')"

if tar cf - $1 | $compress > $archivename ; then
echo "Directory $1 archived as $archivename"
else
echo "Warning: tar encountered errors archiving $1"
fi

exit 0

How It Works
This script is almost all error-checking code, to ensure that it never causes a loss of data or creates an incorrect snapshot. In addition to the typical tests to validate the presence and appropriateness of the starting argument, this script also forces the user to be in the parent directory of the subdirectory to be compressed and archived, which ensures that the archive file is saved in the proper place upon completion. The conditional if [ ! -w . ] ; then verifies that the user has write permission on the current directory. And this script even warns users before archiving if the resultant backup file would be unusually large.

Finally, the actual command that archives the specified directory is

tar cf - $1 | $compress > $archivename

The return code of this command is tested to ensure that the script never deletes the directory if an error of any sort occurs.

Running the Script
This script should be invoked with the name of the desired directory to archive as its only argument. To ensure that the script doesn't try to archive itself, it requires that a subdirectory of the current directory be specified as the argument, rather than ".".

The Results
$ archivedir scripts
Warning: directory scripts is 2224 blocks. Proceed? [n] n
archivedir: archive of directory scripts canceled.

This seemed as though it might be a big archive, so I hesitated to create it, but thinking about it, there's no reason not to proceed after all:

$ archivedir scripts
Warning: directory scripts is 2224 blocks. Proceed? [n] y
Directory scripts archived as scripts.tgz


The results:

$ ls -l scripts.tgz
-rw-r--r-- 1 taylor staff 325648 Jul 14 08:01 scripts.tgz

How to Validating User crontab Entries using Script

One of the most helpful facilities in Unix is cron, with its ability to schedule jobs at arbitrary times in the future, recurring every minute, every few hours, monthly, or annually. Every good system administrator has a Swiss army knife of scripts running from the crontab file.

However, the format for entering cron specifications is a bit tricky, and the cron fields have numeric values, ranges, sets, and even mnemonic names for days of the week or months. What's worse is that the crontab program generates insufficient error messages when scanning in a cron file that might be incorrectly structured.

For example, specify a day of the week with a typo, and crontab reports

"/tmp/crontab.Dj7Tr4vw6R":9: bad day-of-week
crontab: errors in crontab file, can't install

In fact, there's a second error in the sample input file, on line 12, but crontab is going to force us to take the long way around to find it in the script because of its poor error-checking code.

Instead of doing it crontab's way, a somewhat lengthy shell script can step through the crontab files, checking the syntax and ensuring that values are within reasonable ranges. One of the reasons that this validation is possible in a shell script is that sets and ranges can be treated as individual values. So to test whether 3–11 or 4,6,9 are acceptable values for a field, simply test 3 and 11 in the former case, and 4, 6, and 9 in the latter.

The Code
#!/bin/sh

# verifycron - Checks a crontab file to ensure that it's
# formatted properly. Expects standard cron notation of
# min hr dom mon dow CMD
# where min is 0-59, hr is 0-23, dom is 1-31, mon is 1-12 (or names)
# and dow is 0-7 (or names). Fields can be ranges (a-e), lists
# separated by commas (a,c,z), or an asterisk. Note that the step
# value notation of Vixie cron (e.g., 2-6/2) is not supported by this script.


validNum()
{
# Return 0 if valid, 1 if not. Specify number and maxvalue as args
num=$1 max=$2

if [ "$num" = "X" ] ; then
return 0
elif [ ! -s $(echo $num | sed 's/[[:digit:]]//g') ] ; then
return 1
elif [ $num -lt 0 -o $num -gt $max ] ; then
return 1
else
return 0
fi
}

validDay()
{
# Return 0 if a valid dayname, 1 otherwise

case $(echo $1 | tr '[:upper:]' '[:lower:]') in
sun*|mon*|tue*|wed*|thu*|fri*|sat*) return 0 ;;
X) return 0 ;; # special case - it's an "*"
*) return 1
esac
}

validMon()
{
# Return 0 if a valid month name, 1 otherwise

case $(echo $1 | tr '[:upper:]' '[:lower:]') in
jan*|feb*|mar*|apr*|may|jun*|jul*|aug*) return 0 ;;
sep*|oct*|nov*|dec*) return 0 ;;
X) return 0 ;; # special case, it's an "*"
*) return 1 ;;
esac
}

fixvars()
{
# Translate all '*' into 'X' to bypass shell expansion hassles
# Save original input as "sourceline" for error messages

sourceline="$min $hour $dom $mon $dow $command"
min=$(echo "$min" | tr '*' 'X')
hour=$(echo "$hour" | tr '*' 'X')
dom=$(echo "$dom" | tr '*' 'X')
mon=$(echo "$mon" | tr '*' 'X')
dow=$(echo "$dow" | tr '*' 'X')
}

if [ $# -ne 1 ] || [ ! -r $1 ] ; then
echo "Usage: $0 usercrontabfile" >&2; exit 1
fi

lines=0 entries=0 totalerrors=0

while read min hour dom mon dow command
do
lines="$(($lines + 1))"
errors=0

if [ -z "$min" -o "${min%${min#?}}" = "#" ] ; then
continue # nothing to check
elif [ ! -z $(echo ${min%${min#?}} | sed 's/[[:digit:]]//') ] ; then
continue # first char not digit: skip!
fi


entries="$(($entries + 1))"

fixvars

#### Broken into fields, all '*' replaced with 'X'
# Minute check

for minslice in $(echo "$min" | sed 's/[,-]/ /g') ; do
if ! validNum $minslice 60 ; then
echo "Line ${lines}: Invalid minute value \"$minslice\""
errors=1
fi
done

# Hour check
for hrslice in $(echo "$hour" | sed 's/[,-]/ /g') ; do
if ! validNum $hrslice 24 ; then
echo "Line ${lines}: Invalid hour value \"$hrslice\""
errors=1
fi
done

# Day of month check

for domslice in $(echo $dom | sed 's/[,-]/ /g') ; do
if ! validNum $domslice 31 ; then
echo "Line ${lines}: Invalid day of month value \"$domslice\""
errors=1
fi
done

# Month check

for monslice in $(echo "$mon" | sed 's/[,-]/ /g') ; do
if ! validNum $monslice 12 ; then
if ! validMon "$monslice" ; then
echo "Line ${lines}: Invalid month value \"$monslice\""
errors=1
fi
fi
done

# Day of week check

for dowslice in $(echo "$dow" | sed 's/[,-]/ /g') ; do
if ! validNum $dowslice 31 ; then
if ! validDay $dowslice ; then
echo "Line ${lines}: Invalid day of week value \"$dowslice\""
errors=1
fi
fi
done

if [ $errors -gt 0 ] ; then
echo ">>>> ${lines}: $sourceline"
echo ""
totalerrors="$(( $totalerrors + 1 ))"
fi
done < $1

echo "Done. Found $totalerrors errors in $entries crontab entries."

exit 0


How It Works
The greatest challenge in getting this script to work is sidestepping problems with the shell wanting to expand the field value *. An asterisk is perfectly acceptable in a cron entry, and indeed is quite common, but give one to a backtick command and it'll expand to the files in the current directory — definitely not a desired result. Rather than puzzle through the combination of single and double quotes necessary to solve this problem, it proves quite a bit simpler to replace each asterisk with an X, which is what the fixvars function accomplishes.

Also worthy of note is the simple solution to processing comma-and dash-separated lists of values. The punctuation is simply replaced with spaces, and each value is tested as if it were a stand-alone numeric value. That's what the $() sequence does in the for loops:

$(echo "$dow" | sed 's/[,-]/ /g')

With this in the code, it's then simple to step through all numeric values, ensuring that each and every one is valid and within the range for that specific crontab field parameter.

Running the Script
This script is easy to run: Just specify the name of a crontab file as its only argument. To work with an existing crontab file, do this:

$ crontab -l > my.crontab
$ verifycron my.crontab
$ rm my.crontab

The Results
Using a sample crontab file that has two errors and lots of comments, the script produced these results:

$ verifycron sample.crontab
Line 10: Invalid day of week value "Mou"
>>>> 10: 06 22 * * Mou /home/ACeSystem/bin/del_old_ACinventories.pl

Line 12: Invalid minute value "99"
>>>> 12: 99 22 * * 1-3,6 /home/ACeSystem/bin/dump_cust_part_no.pl

Done. Found 2 errors in 17 crontab entries.

The sample crontab file with the two errors, along with all the shell scripts explored in this book, are available at the official Wicked Cool Shell Scripts website, at http://www.intuitive.com/wicked/

Hacking the Script
Two enhancements would be potentially worth adding to this script. Validating the compatibility of month and day combinations would ensure that users don't schedule a cron job to run on, for example, 31 February, which will never happen. It could also be useful to check that the command being invoked can be found, but that would entail parsing and processing a PATH variable (i.e., a list of directories within which to look for commands specified in the script), which can be set explicitly within a crontab file. That could be quite tricky. . . .

How to Kill Processes by Name using script

Linux and some Unixes have a very helpful command called killall, which allows you to kill all running applications that match a specified pattern. It can be quite helpful when you want to kill nine mingetty daemons, or even just to send a SIGHUP signal to xinetd to prompt it to reread its configuration file. Systems that don't have killall can emulate it in a shell script, built around ps for identification of matching processes and kill to send the specified signal.

The tricky part of the script is that the output format from ps varies significantly from OS to OS. For example, consider how differently Mac OS X and Red Hat Linux show running processes in the default ps output:

OSX $ ps
PID TT STAT TIME COMMAND
485 std S 0:00.86 -bash (bash)
581 p2 S 0:00.01 -bash (bash)
RHL9 $ ps
PID TTY TIME CMD
8065 pts/4 00:00:00 bash
12619 pts/4 00:00:00 ps

Worse, rather than model its ps command after a typical Unix command, the GNU ps command accepts BSD-style flags, SYSV-style flags, and GNU-style flags. A complete mishmash!

Fortunately, some of these inconsistencies can be sidestepped in this particular script by using the -cu flag, which produces consistent output that includes the owner of the process, the command name (as opposed to -bash (bash), as in the default Mac OS X output just shown), and the process ID, the lattermost of which is what we're really interested in identifying.

The Code
#!/bin/sh

# killall - Sends the specified signal to all processes that match a
# specific process name.
# By default it only kills processes owned by the same user, unless
# you're root. Use -s SIGNAL to specify a signal to send to the process,
# -u user to specify the user, -t tty to specify a tty,
# and -n to only report what'd be done, rather than doing it.

signal="-INT" # default signal
user="" tty="" donothing=0

while getopts "s:u:t:n" opt; do
case "$opt" in
# Note the trick below: kill wants -SIGNAL but we're asking
# for SIGNAL so we slip the '-' in as part of the assignment
s ) signal="-$OPTARG"; ;;
u ) if [ ! -z "$tty" ] ; then
echo "$0: error: -u and -t are mutually exclusive." >&2
exit 1
fi
user=$OPTARG; ;;
t ) if [ ! -z "$user" ] ; then
echo "$0: error: -u and -t are mutually exclusive." >&2
exit 1
fi
tty=$2; ;;
n ) donothing=1; ;;
? ) echo "Usage: $0 [-s signal] [-u user|-t tty] [-n] pattern" >&2
exit 1
esac
done

shift $(( $OPTIND - 1 ))

if [ $# -eq 0 ] ; then
echo "Usage: $0 [-s signal] [-u user|-t tty] [-n] pattern" >&2
exit 1
fi

if [ ! -z "$tty" ] ; then
pids=$(ps cu -t $tty | awk "/ $1$/ { print \$2 }")
elif [ ! -z "$user" ] ; then
pids=$(ps cu -U $user | awk "/ $1$/ { print \$2 }")
else
pids=$(ps cu -U ${USER:-LOGNAME} | awk "/ $1$/ { print \$2 }")
fi

if [ -z "$pids" ] ; then
echo "$0: no processes match pattern $1" >&2; exit 1
fi

for pid in $pids
do
# Sending signal $signal to process id $pid: kill might
# still complain if the process has finished, the user doesn't
# have permission, etc., but that's okay.
if [ $donothing -eq 1 ] ; then
echo "kill $signal $pid"
else
kill $signal $pid
fi
done

exit 0

How It Works
Because this script is so aggressive, I've put some effort into minimizing false pattern matches, so that a pattern like sh won't match output from ps that contains bash or vi crashtest.c, or other values that embed the pattern. This is done by the pattern-match prefix on the awk command:

awk "/ $1$/ { print \$2 }"

Left-rooting the specified pattern, $1, with a leading space and right-rooting the pattern with a trailing $, causes the script to search for the specified pattern 'sh' in ps output as ' sh$'.

Running the Script
This script has a variety of starting flags that let you modify its behavior. The -s signal flag allows you to specify a signal other than the default interrupt signal, SIGINT, to send to the matching process or processes. The -u user and -t tty flags are useful primarily to the root user in killing all processes associated with a specified user or TTY device, respectively. And the -n flag gives you the option of having the script report what it would do without actually sending any signals. Finally, a pattern must be specified.

The Results
To kill all the csmount processes on my Mac OS X system, I can now use the following:

$ ./killall -n csmount
kill -INT 1292
kill -INT 1296
kill -INT 1306
kill -INT 1310
kill -INT 1318

Hacking the Script
There's an unlikely, though not impossible, bug in this script. To match only the specified pattern, the awk invocation outputs the process ID only of processes that match the pattern plus a leading space that occurs at the end of the input line. However, it's theoretically possible to have two processes running, one called, say, bash and the other emulate bash. If killall is invoked with bash as the pattern, both of these processes will be matched, although only the former is a true match. Solving this to give consistent cross-platform results would prove quite tricky.

If you're motivated, you could also write a script based heavily on the killall script that would let you renice jobs by name, rather than just by process ID. The only change required would be to invoke renice rather than kill.

How to Digging Up Movie Info from IMDb using script

A more sophisticated use of Internet access through lynx and a shell script is demonstrated in this hack, which searches the Internet Movie Database website (http://www.imdb.com/) to find films that match a specified pattern. What makes this script interesting is that it must be able to handle two different formats of return information: If the search pattern matches more than one movie, moviedata returns a list of possible titles, but if there's exactly one movie match, the information about that specific film is returned.

As a result, the script must cache the return information and then search through it once to see if it provides a list of matches and then a second time if it proves to be a summary of the film in question.

The Code
#!/bin/sh

# moviedata - Given a movie title, returns a list of matches, if
# there's more than one, or a synopsis of the movie if there's
# just one. Uses the Internet Movie Database (imdb.com).

imdburl="http://us.imdb.com/Tsearch?restrict=Movies+only&title="
titleurl="http://us.imdb.com/Title?"
tempout="/tmp/moviedata.$$"

summarize_film()
{
# Produce an attractive synopsis of the film

grep "^" $tempout | sed 's/<[^>]*>//g;s/(more)//' <br /> grep '<b class="ch">Plot Outline:</b>' $tempout | \ <br /> sed 's/<[^>]*>//g;s/(more)//;s/(view trailer)//' |fmt|sed 's/^/ /' <br /> exit 0 <br />} <br /> <br />trap "rm -f $tempout" 0 1 15 <br /> <br />if [ $# -eq 0 ] ; then <br /> echo "Usage: $0 {movie title | movie ID}" >&2 <br /> exit 1 <br />fi <br /> <br />fixedname="$(echo $@ | tr ' ' '+')" # for the URL <br /> <br />if [ $# -eq 1 ] ; then <br /> nodigits="$(echo $1 | sed 's/[[:digit:]]*//g')" <br /> if [ -z "$nodigits" ] ; then <br /> lynx -source "$titleurl$fixedname" > $tempout <br /> summarize_film <br /> fi <br />fi <br /> <br />url="$imdburl$fixedname" <br /> <br />lynx -source $url > $tempout <br /> <br />if [ ! -z "$(grep "IMDb title search" $tempout" ] ; then <br /> grep 'HREF="/Title?' $tempout | \ <br /> sed 's/<OL><LI><A HREF="//;s/<\/A><\/LI>//;s/<LI><A HREF="//' | \ <br /> sed 's/">/ -- /;s/<.*//;s/\/Title?//' | \ <br /> sort -u | \ <br /> more <br />else <br /> summarize_film <br />fi <br /> <br />exit 0 <br /> <br />How It Works <br />This script builds a different URL depending on whether the command argument specified is a film name or an IMDb film ID number, and then it saves the lynx output from the web page to the $tempout file. <br /> <br />If the command argument is a film name, the script then examines $tempout for the string "IMDb title search" to see whether the file contains a list of film names (when more than one movie matches the search criteria) or the description of a single film. Using a complex series of sed substitutions that rely on the source code organization of the IMDb site, it then displays the output appropriately for each of those two possible cases. <br /> <br />Running the Script <br />Though short, this script is quite flexible with input formats: You can specify a film title in quotes or as separate words. If more than one match is returned, you can then specify the eight-digit IMDb ID value to select a specific match. <br /> <br />The Results <br />$ moviedata lawrence of arabia <br />0056172 -- Lawrence of Arabia (1962) <br />0099356 -- Dangerous Man: Lawrence After Arabia, A (1990) (TV) <br />0194547 -- With Allenby in Palestine and Lawrence in Arabia (1919) <br />0245226 -- Lawrence of Arabia (1935) <br />0363762 -- Lawrence of Arabia: A Conversation with Steven Spielberg (2000) (V) <br />0363791 -- Making of 'Lawrence of Arabia', The (2000) (V) <br />$ moviedata 0056172 <br />Lawrence of Arabia (1962) <br /> Plot Outline: British lieutenant T.E. Lawrence rewrites the political <br /> history of Saudi Arabia. <br />$ moviedata monsoon wedding <br />Monsoon Wedding (2001) <br /> Plot Outline: A stressed father, a bride-to-be with a secret, a <br /> smitten event planner, and relatives from around the world create <br /> much ado about the preparations for an arranged marriage in India. <br /> <br />Hacking the Script <br />The most obvious hack to this script would be to get rid of the ugly IMDb movie ID numbers. It would be straightforward to hide the movie IDs (because the IDs as shown are rather unfriendly and prone to mistyping) and have the shell script output a simple menu with unique index values (e.g., 1, 2, 3) that can then be typed in to select a particular film. <br /> <br />A problem with this script, as with most scripts that scrape values from a third-party website, is that if IMDb changes its page layout, the script will break and you'll need to rebuild the script sequence. It's a lurking bug, but with a site like IMDb that hasn't changed in years, probably not a dramatic or dangerous one. <br /> <br /></a></a></a></a> <div style='clear: both;'></div> </div> <div class='post-footer'> <div class='post-footer-line post-footer-line-1'> <span class='post-author vcard'> </span> <span class='post-timestamp'> </span> <span class='post-comment-link'> </span> <span class='post-icons'> </span> <div class='post-share-buttons goog-inline-block'> </div> </div> <div class='post-footer-line post-footer-line-2'> <span class='post-labels'> Labels: <a href='https://expertsxchange.blogspot.com/search/label/Linux' rel='tag'>Linux</a>, <a href='https://expertsxchange.blogspot.com/search/label/Mac%20OS' rel='tag'>Mac OS</a>, <a href='https://expertsxchange.blogspot.com/search/label/Unix' rel='tag'>Unix</a> </span> </div> <div class='post-footer-line post-footer-line-3'> <span class='post-location'> </span> </div> </div> </div> </div> <div class='post-outer'> <div class='post hentry uncustomized-post-template' itemprop='blogPost' itemscope='itemscope' itemtype='http://schema.org/BlogPosting'> <meta content='3678427271002128981' itemprop='blogId'/> <meta content='8409273488508145348' itemprop='postId'/> <a name='8409273488508145348'></a> <h3 class='post-title entry-title' itemprop='name'> <a href='https://expertsxchange.blogspot.com/2009/02/how-to-extract-urls-from-web-page-using.html'>How to Extract URLs from a Web Page using Shell Scipt</a> </h3> <div class='post-header'> <div class='post-header-line-1'></div> </div> <div class='post-body entry-content' id='post-body-8409273488508145348' itemprop='description articleBody'> A straightforward shell script application of lynx is to extract a list of URLs on a given web page, which can be quite helpful in a variety of situations.<br /><br />The Code<br />#!/bin/sh<br /><br /># getlinks - Given a URL, returns all of its internal and<br /># external links.<br />if [ $# -eq 0 ] ; then<br /> echo "Usage: $0 [-d|-i|-x] url" >&2<br /> echo "-d=domains only, -i=internal refs only, -x=external only" >&2<br /> exit 1<br />fi<br /><br />if [ $# -gt 1 ] ; then<br /> case "$1" in<br /> -d) lastcmd="cut -d/ -f3 | sort | uniq"<br /> shift<br /> ;;<br /> -i) basedomain="http://$(echo $2 | cut -d/ -f3)/"<br /> lastcmd="grep \"^$basedomain\" | sed \"s|$basedomain||g\" | sort | uniq"<br /> shift<br /> ;;<br /> -x) basedomain="http://$(echo $2 | cut -d/ -f3)/"<br /> lastcmd="grep -v \"^$basedomain\" | sort | uniq"<br /> shift<br /> ;;<br /> *) echo "$0: unknown option specified: $1" >&2; exit 1<br /> esac<br />else<br /> lastcmd="sort | uniq"<br />fi<br /><br />lynx -dump "$1" | \<br /> sed -n '/^References$/,$p' | \<br /> grep -E '[[:digit:]]+\.' | \<br /> awk '{print $2}' | \<br /> cut -d\? -f1 | \<br /> eval $lastcmd<br /><br />exit 0<br /><br />How It Works<br />When displaying a page, lynx shows the text of the page, formatted as best it can, followed by a list of all hypertext references, or links, found on that page. This script simply extracts just the links by using a sed invocation to print everything after the "References" string in the web page text, and then processes the list of links as needed based on the user-specified flags.<br /><br />The one interesting technique demonstrated by this script is the way the variable lastcmd is set to filter the list of links that it extracts according to the flags specified by the user. Once lastcmd is set, the amazingly handy eval command is used to force the shell to interpret the content of the variable as if it were a command, not a variable.<br /><br />Running the Script<br />By default, this script outputs a list of all links found on the specified web page, not just those that are prefaced with http:. There are three optional command flags that can be specified to change the results, however: -d produces just the domain names of all matching URLs, -i produces a list of just the internal references (that is, those references that are found on the same server as the current page), and -x produces just the external references, those URLs that point to a different server.<br /><br />The Results<br />A simple request is a list of all links on a specified website home page:<br /><br />$ getlinks http://www.trivial.net/<br />http://www.intuitive.com/<br />http://www.trivial.net/kudos/index.html<br />http://www.trivial.net/trivial.cgi<br />mailto:nerds@trivial.net<br /><br />Another possibility is to request a list of all domain names referenced at a specific site. This time let's first use the standard Unix tool wc to check how many links are found overall:<br /><br />$ getlinks http://www.amazon.com/ | wc -l<br /> 136<br /><br />Amazon has 136 links on its home page. Impressive! Now, how many different domains does that represent? Let's generate a full list with the -d flag:<br /><br />$ getlinks -d http://www.amazon.com/<br />s1.amazon.com<br />www.amazon.com<br /><br />As you can see, Amazon doesn't tend to point anywhere else. Other sites are different, of course. As an example, here's a list of all external links in my weblog:<br /><br />$ getlinks -x http://www.intuitive.com/blog/<br />LYNXIMGMAP:http://www.intuitive.com/blog/#headermap<br />http://blogarama.com/in.php<br />http://blogdex.media.mit.edu/<br />http://booktalk.intuitive.com/<br />http://chris.pirillo.com/<br />http://cortana.typepad.com/rta/<br />http://dylan.tweney.com/<br />http://fx.crewtags.com/blog/<br />http://geourl.org/near/<br />http://hosting.verio.com/index.php/vps.html<br />http://imajes.info/<br />http://jake.iowageek.com/<br />http://myst-technology.com/mysmartchannels/public/blog/214/<br />http://smattering.org/dryheat/<br />http://www.101publicrelations.com/blog/<br />http://www.APparenting.com/<br />http://www.backupbrain.com/<br />http://www.bloghop.com/<br />http://www.bloghop.com/ratemyblog.htm<br />http://www.blogphiles.com/webring.shtml<br />http://www.blogshares.com/blogs.php<br />http://www.blogstreet.com/blogsqlbin/home.cgi<br />http://www.blogwise.com/<br />http://www.gnome-girl.com/<br />http://www.google.com/search/<br />http://www.icq.com/<br />http://www.infoworld.com/<br />http://www.mail2web.com/<br />http://www.movabletype.org/<br />http://www.nikonusa.com/usa_product/product.jsp<br />http://www.onlinetonight.net/ethos/<br />http://www.procmail.org/<br />http://www.ringsurf.com/netring/<br />http://www.spamassassin.org/<br />http://www.tryingreallyhard.com/<br />http://www.yahoo.com/r/p2 <div style='clear: both;'></div> </div> <div class='post-footer'> <div class='post-footer-line post-footer-line-1'> <span class='post-author vcard'> </span> <span class='post-timestamp'> </span> <span class='post-comment-link'> </span> <span class='post-icons'> </span> <div class='post-share-buttons goog-inline-block'> </div> </div> <div class='post-footer-line post-footer-line-2'> <span class='post-labels'> Labels: <a href='https://expertsxchange.blogspot.com/search/label/Linux' rel='tag'>Linux</a>, <a href='https://expertsxchange.blogspot.com/search/label/Mac%20OS' rel='tag'>Mac OS</a>, <a href='https://expertsxchange.blogspot.com/search/label/Unix' rel='tag'>Unix</a> </span> </div> <div class='post-footer-line post-footer-line-3'> <span class='post-location'> </span> </div> </div> </div> </div> <div class='post-outer'> <div class='post hentry uncustomized-post-template' itemprop='blogPost' itemscope='itemscope' itemtype='http://schema.org/BlogPosting'> <meta content='3678427271002128981' itemprop='blogId'/> <meta content='2433549048411191438' itemprop='postId'/> <a name='2433549048411191438'></a> <h3 class='post-title entry-title' itemprop='name'> <a href='https://expertsxchange.blogspot.com/2009/02/how-to-track-bbc-news-with-lynx-using.html'>How to Track BBC News with lynx using shell Script</a> </h3> <div class='post-header'> <div class='post-header-line-1'></div> </div> <div class='post-body entry-content' id='post-body-2433549048411191438' itemprop='description articleBody'> As I mentioned earlier, one of the unsung heroes of the command-line Internet is unquestionably the lynx web browser (or its newer sibling links). Although you can use it to surf the Web if you dislike graphics, its real power is accessed on the command line itself, within a shell script. <br />The -dump flag, for example, produces the text but not the HTML source, as shown in the following when checking the BBC World Service website, tracking technology news: <br />$ url=http://news.bbc.co.uk/2/low/technology/default.stm <br />$ lynx -dump $url | head <br /> <br /> [1]Skip to main content <br /> BBC NEWS / TECHNOLOGY <br /> [2]Graphics version | [3]Change to UK Edition | [4]BBC Sport Home <br /> _________________________________________________________________ <br /> <br /> [5]News Front Page | [6]Africa | [7]Americas | [8]Asia-Pacific | <br /> [9]Europe | [10]Middle East | [11]South Asia | [12]UK | [13]Business | <br /> [14]Health | [15]Science/Nature | [16]Technology | [17]Entertainment | <br /> [18]Have Your Say <br />This output is not very interesting, but it's easily fed to grep or any other command-line utility, because it's just a text stream at this juncture. Now we can easily check a website to see if there are any stories about a favorite news topic, computer company, or group of people. Let's see if there's any news about games, with a one-line context shown around each match, by using grep: <br />$ lynx -dump $url | grep -C1 -i games <br /> <br /> [21]Screenshot from Vice City [22]Britons' love affair with games <br /> Britain is turning into a nation of keen gamers, research by the UK <br /> games industry trade body suggests. <br /> <br />-- <br /> line-up <br /> Many of the Nintendo games for the Christmas run-up return to familiar <br /> characters and brand names. <br />-- <br /> Virtual pets fed by photos and pronunciation puzzles are just some of <br /> the mobile phone games popular in Japan. <br /> <br /> [28]Next gen consoles spark concern <br /> The next generation of consoles could shake up the games industry, <br /> with smaller firms going bust, say experts. <br />-- <br /> <br /> [37]Text msgs play games with TV <br /> Your TV and mobile are coming closer together, with game shows played <br />-- <br /> [38]Mobile gaming 'set to explode' <br /> Consumers will be spending millions of pounds to play games on their <br /> mobiles by next year, say experts. <br />The numbers in brackets are URL references listed later in the output, so to identify the [37] link, the page needs to be requested again, this time having grep find the associated link URL: <br />$ lynx -dump $url | grep '37\.' <br /> 37. http://news.bbc.co.uk/2/low/technology/3182641.stm <br />Switch to -source rather than -dump, and the output of lynx becomes considerably more interesting. <br />$ lynx -source $url | grep -i 'PublicationDate' <br />""<meta? name= "OriginalPublicationDate" content="2003/08/29 15:01:14" />""?? <br />The -source flag produces the HTML source of the page specified. Pipe that source into a grep or two, and you can extract just about any information from a page, even information within a tag or comment. The bbcnews script that follows lets you easily scrape the top technology stories from the Beeb at any time. <br />The Code <br />#!/bin/sh <br /> <br /># bbcnews - Reports the top stories on the BBC World Service. <br /> <br />url="http://news.bbc.co.uk/2/low/technology/default.stm" <br /> <br />lynx -source $url | \ <br /> sed -n '/Last Updated:/,/newssearch.bbc.co.uk/p' | \ <br /> sed 's/</\ <br /></g;s/>/>\ <br />/g' | \ <br /> grep -v -E '(<|>)' | \ <br /> fmt | \ <br /> uniq <br />How It Works <br />Although this is a short script, it is rather densely packed. These scraper scripts are best built iteratively, looking for patterns to filter in the structure of the web page information and then tuned line by line to produce just the output desired. <br />On the BBC website, this process is surprisingly easy because we're already looking at the low-bandwidth version of the site. The first task is to discard any HTML associated with the navigational menus, bottom material, and so forth, so that we just have the core content of the page, the stories themselves. That's what the first sed does — it reduces the data stream by preserving only the headline and body of the new stories between the "Last Updated" string at the top of the page and the newssearch.bbc.co.uk/p search box at the bottom of the page. <br />The next invocation of sed is uglier than the first, simply because it's doing something peculiar: <br /> sed 's/</\ <br /></g;s/>/>\ <br />/g' <br />Every time it finds an open angle bracket (<), it's replacing it with a carriage return followed by an open angle bracket. Close angle brackets (>) are replaced by a close angle bracket followed by a carriage return. If sed supported an \n notation to specify carriage returns, the second sed invocation would not need to be written across three lines and would read much more easily, as follows: <br />sed 's/</\n</g;s/>/>\n/g' <br />Once the added carriage returns put all the HTML tags on their own lines, the second invocation of grep strips out all the tags (-v inverts the logic of the grep, showing all lines that do not match the pattern, and the -E flag specifies that the argument is a complex regular expression), and the result is fed to fmt to wrap the resultant text lines better. Finally, the uniq command is used to ensure that there aren't multiple blank lines in the output: It removes all nonunique lines from the data stream. <br />Running the Script <br />This script has no arguments, and as long as the BBC hasn't changed its basic low-source page layout, it'll produce a text-only version of the top technology headlines. The first version of the bbcnews script was written around a layout that changed during the summer of 2003: The BBC originally had all its articles wrapped in <div> tags but has since changed it. Fortunately, the update to the script involved only about ten minutes of work. <br />The Results <br />Here's the top technology news at the end of August 2003: <br />$ bbcnews | head -20 <br />Last Updated: Friday, 29 August, 2003, 15:01 GMT 16:01 UK <br /> <br /> Youth suspected of net attack <br /> <br /> An American youth is suspected by the FBI <br /> of being one of the authors of the crippling <br /> MSBlast internet worm, say reports. <br /> <br /> Britons' love affair with games <br /> <br /> Britain is turning into a nation of keen <br /> gamers, research by the UK games industry <br /> trade body suggests. <br /> <br /> Familiar faces in Nintendo's line-up <br /> <br /> Many of the Nintendo games for the Christmas <br /> run-up return to familiar characters and <br /> brand names. <br />Hacking the Script <br />With a little more tuning, you could easily have the top technology story from the BBC News pop up each time you log in to your account. You could also email the results to your mailbox via a cron job every so often, if you wanted: <br />bbcnews | mail -s "BBC Technology News" peter <br />Don't send it to a list, though; there are some copyright and intellectual property issues to consider if you begin republishing Internet content owned by other people. There's a fine line between fair use and violation of copyright, so be thoughtful about what you do with content from another website. <br /> <div style='clear: both;'></div> </div> <div class='post-footer'> <div class='post-footer-line post-footer-line-1'> <span class='post-author vcard'> </span> <span class='post-timestamp'> </span> <span class='post-comment-link'> </span> <span class='post-icons'> </span> <div class='post-share-buttons goog-inline-block'> </div> </div> <div class='post-footer-line post-footer-line-2'> <span class='post-labels'> Labels: <a href='https://expertsxchange.blogspot.com/search/label/Linux' rel='tag'>Linux</a>, <a href='https://expertsxchange.blogspot.com/search/label/Mac%20OS' rel='tag'>Mac OS</a>, <a href='https://expertsxchange.blogspot.com/search/label/Unix' rel='tag'>Unix</a> </span> </div> <div class='post-footer-line post-footer-line-3'> <span class='post-location'> </span> </div> </div> </div> </div> <div class='post-outer'> <div class='post hentry uncustomized-post-template' itemprop='blogPost' itemscope='itemscope' itemtype='http://schema.org/BlogPosting'> <meta content='3678427271002128981' itemprop='blogId'/> <meta content='2250949961481647561' itemprop='postId'/> <a name='2250949961481647561'></a> <h3 class='post-title entry-title' itemprop='name'> <a href='https://expertsxchange.blogspot.com/2009/02/how-to-calculate-time-spent-online.html'>How to Calculate Time Spent Online using shell script</a> </h3> <div class='post-header'> <div class='post-header-line-1'></div> </div> <div class='post-body entry-content' id='post-body-2250949961481647561' itemprop='description articleBody'> <p class="first-para">While every ISP offers relatively expensive unlimited-use dial-up accounts, you might not realize that many ISPs also have very low-cost monthly dial-up accounts if your usage stays below a certain number of hours of connect time in a given month. The problem is, how do you calculate your total connection time on a Unix system? Let's have a look. . . .</p><a name="522"></a><a name="IDX-167"></a> <div class="section"> <h3 class="sect3-title"><a name="523"></a><a name="ch07lev2sec1"></a>The Code</h3> <div class="informalexample"><pre class="literallayout">#!/bin/sh<br /><br /># connecttime - Reports cumulative connection time for month/year entries<br /># found in the system log file. For simplicity, this is an awk program.<br /><br />log="/var/log/system.log" # this is just /var/log/system on some machines<br />tempfile="/tmp/$0.$$"<br /><br />trap "rm $tempfile" 0<br /><br />cat << 'EOF' > $tempfile<br />BEGIN {<br /> lastmonth=""; sum = 0<br />}<br />{<br /> if ( $1 != lastmonth && lastmonth != "" ) {<br /> if (sum > 60) { total = sum/60 " hours" }<br /> else { total = sum " minutes" }<br /> print lastmonth ": " total<br /> sum=0<br /> }<br /> lastmonth=$1<br /> sum += $8<br />}<br />END {<br /> if (sum > 60) { total = sum/60 " hours" }<br /> else { total = sum " minutes" }<br /> print lastmonth ": " total<br />}<br />EOF<br /><br />grep "Connect time" $log | awk -f $tempfile<br /><br />exit 0<br /></pre></div></div> <div class="section"> <h3 class="sect3-title"><a name="524"></a><a name="ch07lev2sec2"></a>How It Works</h3> <p class="first-para">On most Unixes, the system log file contains log entries from the PPP (Point-to-Point Protocol) daemon. Here's an example of a log snippet from a Mac OS X system, looking at <span class="fixed">/var/log/system.log</span>:</p> <div class="informalexample"><pre class="literallayout">$ <b class="bold">grep pppd /var/log/system.log</b><br />Jul 12 10:10:57 localhost pppd[169]: Connection terminated.<br />Jul 12 10:10:57 localhost pppd[169]: Connect time 2.1 minutes.<br />Jul 12 10:10:57 localhost pppd[169]: Sent 15009 bytes, received 387811 bytes.<a name="525"></a><a name="IDX-168"></a><br />Jul 12 10:11:11 localhost pppd[169]: Serial link disconnected.<br />Jul 12 10:11:12 localhost pppd[169]: Exit.<br /></pre></div> <p class="para">There are a number of interesting statistics in this snippet, most importantly the actual connect time. Slice those connect time strings out of the log file, add them up, and you've got your cumulative connect time for the month. This script is smart enough to calculate month-by-month totals even if you don't rotate your logs (though you should; see <a class="chapterjump" href="LiB0068.html#491" target="_parent">Script #55</a>, <i class="emphasis">Rotating Log Files,</i> for details on how to accomplish this quite easily).</p> <p class="para">This script is essentially just a big <span class="fixed">awk</span> program that checks month values in the <span class="fixed">system.log</span> entries to know how to aggregate connect time. When <span class="fixed">$1</span>, the month field in the log file output, is different from <span class="fixed">lastmonth</span>, and <span class="fixed">lastmonth</span> isn't the empty string (which it is when the script begins analyzing the log file), the script outputs the accumulated time for the previous month and resets the accumulator, <span class="fixed">sum</span>, to zero:</p> <div class="informalexample"><pre class="literallayout"> if ( $1 != lastmonth && lastmonth != "" ) {<br /> if (sum > 60) { total = sum/60 " hours" }<br /> else { total = sum " minutes" }<br /> print lastmonth ": " total<br /> sum=0<br /> }<br /></pre></div> <p class="para">The rest of the program should be straightforward reading. Indeed, <span class="fixed">awk</span> programs can be quite clear and readable, which is one reason I like using <span class="fixed">awk</span> for this type of task.</p> <table class="note" border="0" cellpadding="0" cellspacing="0"> <tbody> <tr> <td class="admon-check" valign="top"><br /></td> <td class="admon-title" valign="top" width="144">Handy savings tip </td> <td class="admon-body" valign="top"> <p class="first-para">The dial-up account I use with Earthlink has five hours per month prepaid, so this utility helps ensure that I know when I exceed that and am going to be charged by the hour for additional connect time. It's quite helpful for minimizing those monthly dial-up bills!</p></td></tr></tbody></table></div> <div class="section"> <h3 class="sect3-title"><a name="526"></a><a name="ch07lev2sec3"></a>Running the Script</h3> <p class="first-para">This script has no arguments, though you might need to tweak it to ensure that it's pointing to the log file on your particular system that records <span class="fixed">ppd</span> output messages.</p></div> <div class="section"> <h3 class="sect3-title"><a name="527"></a><a name="ch07lev2sec4"></a>The Results</h3> <p class="first-para">You can tell I don't rotate my log files on my laptop too often:</p> <div class="informalexample"><pre class="literallayout">$ <b class="bold">connecttime</b><br />Apr: 4.065 hours<br />Jun: 26.71 hours<br />Jul: 1.96333 hours<br />Aug: 15.085 hours<br /></pre></div></div> <div style='clear: both;'></div> </div> <div class='post-footer'> <div class='post-footer-line post-footer-line-1'> <span class='post-author vcard'> </span> <span class='post-timestamp'> </span> <span class='post-comment-link'> </span> <span class='post-icons'> </span> <div class='post-share-buttons goog-inline-block'> </div> </div> <div class='post-footer-line post-footer-line-2'> <span class='post-labels'> Labels: <a href='https://expertsxchange.blogspot.com/search/label/Linux' rel='tag'>Linux</a>, <a href='https://expertsxchange.blogspot.com/search/label/Mac%20OS' rel='tag'>Mac OS</a>, <a href='https://expertsxchange.blogspot.com/search/label/Unix' rel='tag'>Unix</a> </span> </div> <div class='post-footer-line post-footer-line-3'> <span class='post-location'> </span> </div> </div> </div> </div> <div class='post-outer'> <div class='post hentry uncustomized-post-template' itemprop='blogPost' itemscope='itemscope' itemtype='http://schema.org/BlogPosting'> <meta content='3678427271002128981' itemprop='blogId'/> <meta content='6460763127902392903' itemprop='postId'/> <a name='6460763127902392903'></a> <h3 class='post-title entry-title' itemprop='name'> <a href='https://expertsxchange.blogspot.com/2009/02/how-to-producing-summary-listings-of.html'>How to Producing Summary Listings of iTunes Libraries using script</a> </h3> <div class='post-header'> <div class='post-header-line-1'></div> </div> <div class='post-body entry-content' id='post-body-6460763127902392903' itemprop='description articleBody'> If you've used the excellent Mac OS X application iTunes for any length of time, you're sure to have a massive playlist of CDs that you've scanned, downloaded, swapped, or what-have-you. Unfortunately, for all its wonderful capabilities, iTunes doesn't have an easy way to export a list of your music in a succinct and easy-to-read format. Fortunately, it's not hard to write a script to offer this functionality.<br /><br />The Code<br />#!/bin/sh<br /><br /># itunelist - Lists your iTunes library in a succinct and attractive<br /># manner, suitable for sharing with others, or for<br /># synchronizing (with diff) iTune libraries on different<br /># computers and laptops.<br /><br />itunehome="$HOME/Music/iTunes"<br />ituneconfig="$itunehome/iTunes Music Library.xml"<br /><br />musiclib="/$(grep '>Music Folder<' "$ituneconfig" | cut -d/ -f5- | \<br /> cut -d\< -f1 | sed 's/%20/ /g')"<br /><br />echo "Your music library is at $musiclib"<br /><br />if [ ! -d "$musiclib" ] ; then<br /> echo "$0: Confused: Music library $musiclib isn't a directory?" >&2<br /> exit 1<br />fi<br /><br />exec find "$musiclib" -type d -mindepth 2 -maxdepth 2 \! -name '.*' -print |<br /> sed "s|$musiclib/||"<br /><br />How It Works<br />Like many modern computer applications, iTunes expects its music library to be in a standard location — in this case ~/Music/iTunes Music Library/iTunes Library/ — but allows you to move it elsewhere if desired. The script needs to be able to ascertain the different location, and that's done by extracting the Music Folder field value from the iTunes preferences file. That's what this pipe accomplishes:<br /><br />musiclib="/$(grep '>Music Folder<' "$ituneconfig" | cut -d/ -f5- | \<br /> cut -d\< -f1 | sed 's/%20/ /g')"<br /><br />The preferences file ($ituneconfig) is an XML data file, so it's necessary to do some chopping to identify the exact Music Folder field value. Here's what the Music Folder value in my own iTunes config file looks like:<br /><br />file://localhost/Volumes/110GB/iTunes%20Library/<br /><br />The Music Folder value is actually stored as a fully qualified URL, interestingly enough, so we need to chop off the file://localhost/ prefix, which is the job of the first cut command. Finally, because many directories in Mac OS X include spaces, and because the Music Folder field is saved as a URL, all spaces in that field are mapped to %20 sequences and have to be restored to spaces by the sed invocation before proceeding.<br /><br />With the Music Folder name determined, it's now easy to generate music lists on two Macintosh systems (or even an iPod!) and then use the diff command to compare them, making it a breeze to see which albums are unique to one or the other system and perhaps to sync them up.<br /><br />Running the Script<br />There are no command arguments or flags to this script.<br /><br />The Results<br />$ itunelist | head<br />Your music library is at /Volumes/110GB/iTunes Library/<br />Acoustic Alchemy/Blue Chip<br />Acoustic Alchemy/Red Dust & Spanish Lace<br />Acoustic Alchemy/Reference Point<br />Adrian Legg/Mrs. Crowe's Blue Waltz<br />Al Jarreau/Heaven And Earth<br />Alan Parsons Project/Best Of The Alan Parsons Project<br />Alan Parsons Project/Eve<br />Alan Parsons Project/Eye In The Sky<br />Alan Parsons Project/I Robot<br /><br />Hacking the Script<br />All right, this isn't about hacking the script per se, but because the iTunes library directory is saved as a fully qualified URL, it would be most interesting to experiment with having a web-accessible iTunes directory and then using the URL of that directory as the Music Folder value in the XML file.... <div style='clear: both;'></div> </div> <div class='post-footer'> <div class='post-footer-line post-footer-line-1'> <span class='post-author vcard'> </span> <span class='post-timestamp'> </span> <span class='post-comment-link'> </span> <span class='post-icons'> </span> <div class='post-share-buttons goog-inline-block'> </div> </div> <div class='post-footer-line post-footer-line-2'> <span class='post-labels'> Labels: <a href='https://expertsxchange.blogspot.com/search/label/Linux' rel='tag'>Linux</a>, <a href='https://expertsxchange.blogspot.com/search/label/Mac%20OS' rel='tag'>Mac OS</a>, <a href='https://expertsxchange.blogspot.com/search/label/Unix' rel='tag'>Unix</a> </span> </div> <div class='post-footer-line post-footer-line-3'> <span class='post-location'> </span> </div> </div> </div> </div> <div class='post-outer'> <div class='post hentry uncustomized-post-template' itemprop='blogPost' itemscope='itemscope' itemtype='http://schema.org/BlogPosting'> <meta content='3678427271002128981' itemprop='blogId'/> <meta content='3762834503140190981' itemprop='postId'/> <a name='3762834503140190981'></a> <h3 class='post-title entry-title' itemprop='name'> <a href='https://expertsxchange.blogspot.com/2009/02/managing-apache-passwords-using-shell.html'>Managing Apache Passwords using shell script</a> </h3> <div class='post-header'> <div class='post-header-line-1'></div> </div> <div class='post-body entry-content' id='post-body-3762834503140190981' itemprop='description articleBody'> One terrific feature of the Apache web server is that it offers built-in support for password-protected directories, even on a shared public server. It's a great way to have private, secure, and limited-access information on your website, whether you have a pay subscription service or you just want to ensure that family pictures are viewed only by family.<br /><br />Standard configurations require that in the password-protected directory you manage a data file called .htaccess, which specifies the security "zone" name and, most importantly, points to a separate data file, which in turn contains the account name and password pairs that are used to validate access to the directory. Managing this file is not a problem, except that the only tool included with Apache for doing so is the primitive htpasswd program, which is run on the command line. Instead, this script, apm, one of the most complex and sophisticated scripts in this book, offers a password management tool that runs as a CGI script and lets you easily add new accounts, change the passwords on existing accounts, and delete accounts from the access list.<br /><br />To get started, you will need a properly formatted .htaccess file to control access to the directory it's located within. For demonstration purposes, this file might look like the following:<br /><br />$ cat .htaccess<br />AuthUserFile /web/intuitive/wicked/examples/protected/.htpasswd<br />AuthGroupFile /dev/null<br />AuthName "Sample Protected Directory"<br />AuthType Basic<br /><br /><limit get=""><br />require valid-user<br /></limit><br /><br />A separate file, .htpasswd, contains all the account and password pairs. If this file doesn't yet exist, you'll need to create one, but a blank one is fine: Use touch.htpasswd and ensure that it's writable by the user ID that runs Apache itself (probably user nobody). Then we're ready for the script.<br /><br />The Code<br />#!/bin/sh<br /><br /># apm - Apache Password Manager. Allows the administrator to easily<br /># manage the addition, update, or deletion of accounts and passwords<br /># for access to a subdirectory of a typical Apache configuration (where<br /># the config file is called .htaccess).<br /><br />echo "Content-type: text/html"<br />echo ""<br />echo "<title>Apache Password Manager Utility"

myname="$(basename $0)"
temppwfile="/tmp/apm.$$"; trap "/bin/rm -f $temppwfile" 0
footer="apm-footer.html"
htaccess=".htaccess" # if you use a /cgi-bin, make sure this points
# to the correct .htaccess file!

# Modern versions of 'htpasswd' include a -b flag that lets you specify
# the password on the command line. If yours can do that, specify it
# here, with the '-b' flag:
# htpasswd="/usr/local/bin/htpasswd -b"
# Otherwise, there's a simple Perl rewrite of this script that is a good
# substitute, at http://www.intuitive.com/shellhacks/examples/httpasswd-b.pl

htpasswd="/web/intuitive/wicked/examples/protected/htpasswd-b.pl"
if [ "$REMOTE_USER" != "admin" -a -s $htpasswd ] ; then
echo "Error: you must be user admin to use APM."
exit 0
fi

# Now get the password filename from the .htaccess file

if [ ! -r "$htaccess" ] ; then
echo "Error: cannot read $htaccess file in this directory."
exit 0
fi

passwdfile="$(grep "AuthUserFile" $htaccess | cut -d\ -f2)"

if [ ! -r $passwdfile ] ; then
echo "Error: can't read password file: can't make updates."
exit 0
elif [ ! -w $passwdfile ] ; then
echo "Error: can't write to password file: can't update."
exit 0
fi

echo "

Apache Password Manager

"

action="$(echo $QUERY_STRING | cut -c3)"
user="$(echo $QUERY_STRING|cut -d\& -f2|cut -d= -f2|tr '[:upper:]' '[:lower:]')"

case "$action" in
A ) echo "

Adding New User $user

"
if [ ! -z "$(grep -E "^${user}:" $passwdfile)" ] ; then
echo "Error: user $user already appears in the file."
else
pass="$(echo $QUERY_STRING|cut -d\& -f3|cut -d= -f2)"
if [ ! -z "$(echo $pass | tr -d '[[:upper:][:lower:][:digit:]]')" ]
then
echo "Error: passwords can only contain a-z A-Z 0-9 ($pass)"
else
$htpasswd $passwdfile $user $pass
echo "Added!
"
fi
fi
;;
U ) echo "

Updating Password for user $user

"
if [ -z "$(grep -E "^${user}:" $passwdfile)" ] ; then
echo "Error: user $user isn't in the password file?"
echo "
";cat $passwdfile;echo "
"
echo "searched for "^${user}:" in $passwdfile"
else
pass="$(echo $QUERY_STRING|cut -d\& -f3|cut -d= -f2)"
if [ ! -z "$(echo $pass | tr -d '[[:upper:][:lower:][:digit:]]')" ]
then
echo "Error: passwords can only contain a-z A-Z 0-9 ($pass)"
else
grep -vE "^${user}:" $passwdfile > $temppwfile
mv $temppwfile $passwdfile
$htpasswd $passwdfile $user $pass
echo "Updated!
"
fi
fi
;;
D ) echo "

Deleting User $user

"
if [ -z "$(grep -E "^${user}:" $passwdfile)" ] ; then
echo "Error: user $user isn't in the password file?"
elif [ "$user" = "admin" ] ; then
echo "Error: you can't delete the 'admin' account."
else
grep -vE "^${user}:" $passwdfile > $temppwfile
mv $temppwfile $passwdfile
echo "Deleted!
"
fi
;;
esac

# Always list the current users in the password file...

echo "

"
echo ""
oldIFS=$IFS ; IFS=":" # change word split delimiter
while read acct pw ; do
echo ""
done < $passwdfile echo "
List "
echo "of all current users
$acct"
echo "[delete]
"
IFS=$oldIFS # and restore it

# Build optionstring with all accounts included
optionstring="$(cut -d: -f1 $passwdfile | sed 's/^//'|tr '\n' ' ')"

# And output the footer
sed -e "s/--myname--/$myname/g" -e "s/--options--/$optionstring/g" < $footer exit 0 How It Works There's a lot working together for this script to function. Not only do you need to have your Apache configuration (or equivalent) correct, but you need to have the correct entries in the .htaccess file and you need an .htpasswd file with (ideally) at least an entry for the admin user. The script itself extracts the htpasswd filename from the .htaccess file and does a variety of tests to sidestep common htpasswd error situations, including an inability for the script to write to the file. It also checks to ensure that the user is logged in as admin if the password file exists and is nonzero in size. All of this occurs before the main block of the script, the case statement. Processing Changes to .htpasswd The case statement ascertains which of three possible actions is requested (A = add a user, U = update a user record, and D = delete a user) and invokes the correct segment of code accordingly. The action and the user account on which to perform the action are specified in the QUERY_STRING variable (sent by the web browser to the server) as a=X&u=Y, where X is the action letter code and Y is the specified username. When a password is being changed or a user is being added, a third argument, p, is needed and sent to the script. For example, let's say I was adding a new user called joe, with the password knife. This action would result in the following QUERY_STRING being given to the script from the web server: a=A&u=joe&p=knife The script would unwrap this so that action was A, user was joe, and pass was knife. Then it would ensure that the password contains only valid alphabetic characters in the following test: if [ ! -z "$(echo $pass | tr -d '[[:upper:][:lower:][:digit:]]')" ] ; then echo "Error: passwords can only contain a-z A-Z 0-9 ($pass)" Finally, if all was well, it would invoke the htpasswd program to encrypt the password and add the new entry to the .htpasswd file: $htpasswd $passwdfile $user $pass Listing All User Accounts In addition to processing requested changes to the .htpasswd file, directly after the case statement this script also produces an HTML table that lists each user in the .htpasswd file, along with a [delete] link. After producing three lines of HTML output for the heading of the table, the script continues with the interesting code: oldIFS=$IFS ; IFS=":" # change word split delimiter while read acct pw ; do echo "$acct"
echo "[delete]
"
done < $passwdfile echo "" IFS=$oldIFS # and restore it This while loop reads the name and password pairs from the .htpasswd file through the trick of changing the input field separator (IFS) to a colon (and changing it back when done). Adding a Footer of Actions to Take The script also relies on the presence of an HTML file called apm-footer.html that contains quite a bit of code itself, including occurrences of the strings "--myname--" and "--options--", which are replaced by the current name of the CGI script and the list of users, respectively, as the file is output to stdout. sed -e "s/--myname--/$myname/g" -e "s/--options--/$optionstring/g" < $footer The $myname variable is processed by the CGI engine, which replaces the variable with the actual name of the script. The script itself builds the $optionstring variable from the account name and password pairs in the .htpasswd file: optionstring="$(cut -d: -f1 $passwdfile | sed 's/^//'|tr '\n' ' ')" And here's the HTML footer file itself, which provides the ability to add a user, update a user's password, and delete a user:





Password Manager Actions





add user:


password:








update


password:




delete









Running the Script
You'll most likely want to have this script in the same directory you're endeavoring to protect with passwords, although you can also put it in your cgi-bin directory: Just tweak the htpasswd value at the beginning of the script as appropriate. You'll also need an .htaccess file defining access permissions and an .htpasswd file that's at least zero bytes and writable, if nothing else.

Very helpful tip When you use apm, make sure that the first account you create is admin, so you can use the script upon subsequent invocations! There's a special test in the code that allows you to create the admin account if .htpasswd is empty.


The Result
The result of running the apm script is shown in Figure 9-1. Notice in the screen shot that it not only lists all the accounts, with a delete link for each, but also, in the bottom section, offers options for adding another account, changing the password of an existing account, deleting an account, or listing all the accounts.


A State Capitals Quiz using shell script

Once you have a tool for choosing a line randomly from a file, Displaying Random Text, there's no limit to the type of quiz games you can write. In this instance, I've pulled together a list of the capitals of all 50 states in the United States of America; this script randomly chooses one, shows the state, and asks the user to type in the matching capital.

The Code
#!/bin/sh

# states - A state capital guessing game. Requires the state capitals
# data file at http://www.intuitive.com/wicked/examples/state.capitals.txt.

db="/usr/lib/games/state.capitals.txt"
randomquote="$HOME/bin/randomquote.sh" # Script #76

if [ ! -r $db ] ; then
echo "$0: Can't open $db for reading." >&2
echo "(get http://www.intuitive.com/wicked/examples/state.capitals.txt" >&2
echo "save the file as $db and you're ready to play!)" >&2
exit 1
fi

guesses=0; correct=0; total=0

while [ "$guess" != "quit" ] ; do
thiskey="$($randomquote$db)"

state="$(echo $thiskey | cut -d\ -f1 | sed 's/-/ /g')"
city="$(echo $thiskey | cut -d\ -f2 | sed 's/-/ /g')"
match="$(echo $city | tr '[:upper:]' '[:lower:]')"

guess="??" ; total=$(( $total + 1 )) ;

echo ""
echo "What city is the capital of $state?"

while [ "$guess" != "$match" -a "$guess" != "next" -a "$guess" != "quit" ]
do
echo -n "Answer: "
read guess

if [ "$guess" = "$match" -o "$guess" = "$city" ] ; then
echo ""
echo "*** Absolutely correct! Well done! ***"
correct=$(( $correct + 1 ))
guess=$match
elif [ "$guess" = "next" -o "$guess" = "quit" ] ; then
echo ""
echo "$city is the capital of $state."
else
echo "I'm afraid that's not correct."
fi
done

done

echo "You got $correct out of $total presented."
exit 0

How It Works
For such an entertaining game, states is very simple scripting. The data file contains state/capital pairs, with all spaces in the state and capital names replaced with dashes and the two fields separated by a single space. As a result, extracting the city and state names from the data is easy:

state="$(echo $thiskey | cut -d\ -f1 | sed 's/-/ /g')"
city="$(echo $thiskey | cut -d\ -f2 | sed 's/-/ /g')"


Each guess is compared against both the all-lowercase version of the city name (match) and the actual correctly capitalized city name to see if it's correct. If not, the guess is compared against the two command words next and quit. If either matches, the script shows the answer and either prompts for another state or quits, as appropriate.

Running the Script
This script has no arguments or command flags.

The Results
Ready to quiz yourself on state capitals?

$ states

What city is the capital of Indiana?
Answer: Bloomington
I'm afraid that's not correct.
Answer: Indianapolis

*** Absolutely correct! Well done! ***

What city is the capital of Massachusetts?
Answer: Boston

*** Absolutely correct! Well done! ***

What city is the capital of West Virginia?
Answer: Charleston

*** Absolutely correct! Well done! ***

What city is the capital of Alaska?
Answer: Fairbanks
I'm afraid that's not correct.
Answer: Anchorage
I'm afraid that's not correct.
Answer: Nome
I'm afraid that's not correct.
Answer: Juneau

*** Absolutely correct! Well done! ***

What city is the capital of Oregon?
Answer: quit

Salem is the capital of Oregon.
You got 4 out of 5 presented.


Fortunately, the game tracks only ultimately correct guesses, not how many incorrect guesses you made or whether you popped over to Google to get the correct answer! :-)

Hacking the Script
Probably the greatest weakness in this game is that it's so picky about spelling. A useful modification would be to add some code to allow fuzzy matching, so that the user entry of Juneu might match Juneau, for example. This could be done using a modified Soundex algorithm, in which all vowels are removed and all doubled letters are squished down to a single letter (e.g., Annapolis would transform to npls). This might be too forgiving for your tastes, but the general concept is worth considering.

As with other games, a "hint" function would be useful too. Perhaps it would show the first letter of the correct answer when requested but keep track of how many hints were used as the play proceeded.

Although this game is written around state capitals, it would be quite trivial to modify the script to work with any sort of paired data file. For example, you could create an Italian vocabulary quiz with a slightly different file, or a country/ currency match, or even a politician/political party quiz. Again, as we've seen repeatedly in Unix, writing something that is reasonably general purpose allows it to be reused in useful and occasionally unexpected ways.

Guess the Word Before It's Too Late: Hangman Fun Game using shell script

A classic word game with a macabre metaphor, hangman is nonetheless popular and enjoyable. In the game, you guess letters that might be in the hidden word, and each time you guess incorrectly, the man hanging on the gallows has an additional body part drawn in. Make too many wrong guesses, and the man is fully illustrated, so not only do you lose, but, well, you presumably die too. Not very pleasant!

However, the game itself is fun, and writing it as a shell script proves surprisingly easy.

The Code
#!/bin/sh

# hangman - A rudimentary version of the hangman game. Instead of showing a
# gradually embodied hanging man, this simply has a bad guess countdown.
# You can optionally indicate the initial distance from the gallows as the only
# arg.

wordlib="/usr/lib/games/long-words.txt"
randomquote="$HOME/bin/randomquote.sh" # Script #76
empty="\." # we need something for the sed [set] when $guessed=""
games=0

if [ ! -r $wordlib ] ; then
echo "$0: Missing word library $wordlib" >&2
echo "(online: http://www.intuitive.com/wicked/examples/long-words.txt" >&2
echo "save the file as $wordlib and you're ready to play!)" >&2
exit 1
fi

while [ "$guess" != "quit" ] ; do
match="$($randomquote $wordlib)" # pick a new word from the library

if [ $games -gt 0 ] ; then
echo ""
echo "*** New Game! ***"
fi

games="$(($games + 1))"
guessed="" ; guess="" ; bad=${1:-6}
partial="$(echo $match | sed "s/[^$empty${guessed}]/-/g")"

while [ "$guess" != "$match" -a "$guess" != "quit" ] ; do

echo ""
if [ ! -z "$guessed" ] ; then
echo -n "guessed: $guessed, "
fi
echo "steps from gallows: $bad, word so far: $partial"

echo -n "Guess a letter: "
read guess
echo ""

if [ "$guess" = "$match" ] ; then
echo "You got it!"
elif [ "$guess" = "quit" ] ; then
sleep 0 # a 'no op' to avoid an error message on 'quit'
elif [ $(echo $guess | wc -c | sed 's/[^[:digit:]]//g') -ne 2 ] ; then
echo "Uh oh: You can only guess a single letter at a time"
elif [ ! -z "$(echo $guess | sed 's/[[:lower:]]//g')" ] ; then
echo "Uh oh: Please only use lowercase letters for your guesses"
elif [ -z "$(echo $guess | sed "s/[$empty$guessed]//g")" ] ; then
echo "Uh oh: You have already tried $guess"
elif [ "$(echo $match | sed "s/$guess/-/g")" != "$match" ] ; then
guessed="$guessed$guess"
partial="$(echo $match | sed "s/[^$empty${guessed}]/-/g")"
if [ "$partial" = "$match" ] ; then
echo "** You've been pardoned!! Well done! The word was \"$match\"."
guess="$match"
else
echo "* Great! The letter \"$guess\" appears in the word!"
fi
elif [ $bad -eq 1 ] ; then
echo "** Uh oh: you've run out of steps. You're on the platform... "
echo "** The word you were trying to guess was \"$match\""
guess="$match"
else
echo "* Nope, \"$guess\" does not appear in the word."
guessed="$guessed$guess"
bad=$(($bad - 1))
fi
done
done
exit 0

How It Works
The tests in this script are all interesting and worth examination. Consider this test to see if the player has entered more than a single letter as his or her guess:

elif [ $(echo $guess | wc -c | sed 's/[^[:digit:]]//g') -ne 2 ] ; then


Why test for the value 2 rather than 1? Because the entered value has a carriage return appended by the read statement, and so it has two letters if it's correct, not one. The sed in this statement strips out all nondigit values, of course, to avoid any confusion with the leading tab that wc likes to emit.

Testing for lowercase is straightforward: Remove all lowercase letters from guess and see if the result is zero (empty) or not:

elif [ ! -z "$(echo $guess | sed 's/[[:lower:]]//g')" ] ; then

And, finally, to see if the user has guessed the letter already, transform the guess such that any letters in guess that also appear in the guessed variable are removed, and see if the result is zero (empty) or not:

elif [ -z "$(echo $guess | sed "s/[$empty$guessed]//g")" ] ; then

Apart from all these tests, however, the trick behind getting hangman to work is to translate into dashes all occurrences in the original word of each guessed letter and then to compare the result to the original word. If they're different, the guessed letter is in that word:

elif [ "$(echo $match | sed "s/$guess/-/g")" != "$match" ] ; then

One of the key ideas that made it possible to write hangman was that the partially filled-in word shown to the player, the variable partial, is rebuilt each time a correct guess is made. Because the variable guessed accumulates each letter guessed by the player, a sed transformation that translates into a dash each letter in the original word that is not in the guessed string does the trick:

partial="$(echo $match | sed "s/[^$empty${guessed}]/-/g")"

Running the Script
The hangman game has one optional argument: If you specify a numeric value as a parameter, it will use that as the number of incorrect guesses allowed, rather than the default of 6.

The Results
$ hangman

steps from gallows: 6, word so far: -------------
Guess a letter: e

* Great! The letter "e" appears in the word!

guessed: e, steps from gallows: 6, word so far: -e--e--------
Guess a letter: i
* Great! The letter "i" appears in the word!

guessed: ei, steps from gallows: 6, word so far: -e--e--i-----
Guess a letter: o

* Great! The letter "o" appears in the word!

guessed: eio, steps from gallows: 6, word so far: -e--e--io----
Guess a letter: u

* Great! The letter "u" appears in the word!

guessed: eiou, steps from gallows: 6, word so far: -e--e--iou---
Guess a letter: m

* Nope, "m" does not appear in the word.

guessed: eioum, steps from gallows: 5, word so far: -e--e--iou---
Guess a letter: n

* Great! The letter "n" appears in the word!

guessed: eioumn, steps from gallows: 5, word so far: -en-en-iou---
Guess a letter: r

* Nope, "r" does not appear in the word.

guessed: eioumnr, steps from gallows: 4, word so far: -en-en-iou---
Guess a letter: s

* Great! The letter "s" appears in the word!

guessed: eioumnrs, steps from gallows: 4, word so far: sen-en-ious--
Guess a letter: t

* Great! The letter "t" appears in the word!

guessed: eioumnrst, steps from gallows: 4, word so far: sententious--
Guess a letter: l

* Great! The letter "l" appears in the word!

guessed: eioumnrstl, steps from gallows: 4, word so far: sententiousl-
Guess a letter: y

** You've been pardoned!! Well done! The word was "sententiously".

*** New Game! ***
steps from gallows: 6, word so far: ----------
Guess a letter: quit

Hacking the Script
Obviously it's quite difficult to have the fancy guy-hanging-on-the-gallows graphic if we're working with a shell script, so we use the alternative of counting "steps to the gallows" instead. If you were motivated, however, you could probably have a series of predefined "text" graphics, one for each step, and output them as the game proceeds. Or you could choose a nonviolent alternative of some sort, of course!

Note that it is possible to pick the same word twice, but with the default word list containing 2,882 different words, there's not much chance of that occurring. If this is a concern, however, the line where the word is chosen could also save all previous words in a variable and screen against them to ensure that there aren't any repeats.

Finally, if you were motivated, it'd be nice to have the guessed letters list be sorted alphabetically. There are a couple of approaches to this, but I think I'd try to use sed|sort.

How to create Unscramble: A Word Game using shell script

If you've seen the Jumble game in your newspaper or played word games at all, you're familiar with the basic concept of this game: A word is picked at random and then scrambled. Your task is to figure out and guess what the original word is in the minimum number of turns.

The Code
#!/bin/sh

# unscramble - Picks a word, scrambles it, and asks the user to guess
# what the original word (or phrase) was.

wordlib="/usr/lib/games/long-words.txt"
randomquote="$HOME/bin/randomquote" # Script #76

scrambleword()
{
# Pick a word randomly from the wordlib, and scramble it.
# Original word is $match and scrambled word is $scrambled

match="$($randomquote $wordlib)"

echo "Picked out a word!"

len=$(echo $match | wc -c | sed 's/[^[:digit:]]//g')
scrambled=""; lastval=1

for ((val=1; $val < $len ;))
do
if [ $(perl -e "print int rand(2)") -eq 1 ] ; then
scrambled=$scrambled$(echo $match | cut -c$val)
else
scrambled=$(echo $match | cut -c$val)$scrambled
fi
val=$(($val + 1))
done
}

if [ ! -r $wordlib ] ; then
echo "$0: Missing word library $wordlib" >&2
echo "(online: http://www.intuitive.com/wicked/examples/long-words.txt" >&2
echo "save the file as $wordlib and you're ready to play!)" >&2
exit 1
fi

newgame=""; guesses=0; correct=0; total=0

until [ "$guess" = "quit" ] ; do

scrambleword

echo ""
echo "You need to unscramble: $scrambled"
guess="??" ; guesses=0
total=$(($total + 1))

while [ "$guess" != "$match" -a "$guess" != "quit" -a "$guess" != "next" ]
do
echo ""
echo -n "Your guess (quit|next) : "
read guess

if [ "$guess" = "$match" ] ; then
guesses=$(($guesses + 1))
echo ""
echo "*** You got it with tries = ${guesses}! Well done!! ***"
echo ""
correct=$(($correct + 1))
elif [ "$guess" = "next" -o "$guess" = "quit" ] ; then
echo "The unscrambled word was \"$match\". Your tries: $guesses"
else
echo "Nope. That's not the unscrambled word. Try again."
guesses=$(($guesses + 1))
fi
done
done

echo "Done. You correctly figured out $correct out of $total scrambled words."

exit 0

How It Works
To randomly pick a single line from a file, this script uses Script #76, Displaying Random Text, even though it was originally written to work with web pages. Like many good Unix utilities, it turns out to be a useful building block in other contexts than the one it was intended for:

match="$($randomquote $wordlib)"

The toughest part of this script was figuring out how to scramble a word. There's no handy Unix utility for that, but fortunately it turns out that if we assemble the scrambled word by going letter by letter through the correctly spelled word and randomly adding each subsequent letter to the scrambled sequence at either the beginning or the end of the sequence, we quite effectively scramble the word differently and unpredictably each time:

if [ $(perl -e "print int rand(2)") -eq 1 ] ; then
scrambled=$scrambled$(echo $match | cut -c$val)
else
scrambled=$(echo $match | cut -c$val)$scrambled
fi

Notice where $scrambled is located in the two lines: In the first line the added letter is appended, while in the second it is prepended.

Otherwise the main game logic should be easily understood: The outer while loop runs until the user enters quit as a guess, while the inner loop runs until the user either guesses the word or types next to skip to the next word.

Running the Script
This script has no arguments or parameters, so just type in the name, and you're ready to play!

The Results
$ unscramble
Picked out a word!

You need to unscramble: ninrenoccg

Your guess (quit|next) : concerning

*** You got it with tries = 1! Well done!! ***

Picked out a word!

You need to unscramble: esivrmipod

Your guess (quit|next) : quit
The unscrambled word was "improvised". Your tries: 0
Done. You correctly figured out 1 out of 2 scrambled words.

Clearly an inspired guess on that first one!

Hacking the Script
Perhaps some method of offering a clue would make this game more interesting or, alternatively, a flag that requests the minimum word length that is acceptable. To accomplish the former, perhaps the first n letters of the unscrambled word could be shown for a certain penalty in the scoring; each clue requested would show one additional letter. For the latter, you'd need to have an expanded word dictionary, as the one included with the script has a minimum word length of ten letters, which makes it rather tricky!

How to Set Terminal Title Dynamically using script

This is a fun little script for Mac OS X users who like to work in the Terminal application. Instead of having to use the Terminal > Window Settings > Window dialog box to set or change the window title, you can use this script to change it whenever you like.

The Code
#! /bin/sh

# titleterm - Tells the Mac OS X Terminal application to change its title
# to the value specified as an argument to this succinct script.

if [ $# -eq 0 ]; then
echo "Usage: $0 title" >&2
exit 1
else
echo -ne "\033]0;$1\007"
fi

exit 0

How It Works
The Terminal application has a variety of different secret escape codes that it understands, and the titleterm script sends a sequence of ESC ] 0; title BEL, which changes the title to the specified value.

Running the Script
To change the title of the Terminal window, simply type in the new title you desire.

The Results
There's no apparent output from the command:

$ titleterm $(pwd)


However, it instantly changes the title of the Terminal window to the present working directory.

Hacking the Script
With one small addition to your .cshrc or .bashrc (depending on what login shell you have), you can automatically have the Terminal window title always show the current working directory. To use this to show your current working directory, for example, you can use either of the following:

alias precmd 'titleterm "$PWD"' [tcsh]

export PROMPT_COMMAND="titleterm \"\$PWD\"" [bash]

If you run either the tcsh shell (the default login shell for 10.2.x) or the bash shell (the default shell for 10.3.x, the so-called Panther release of Mac OS X), you can drop one of the commands above into your .cshrc or .bashrc, and, starting the next time you open up a Terminal window, you'll find that your window title changes each time you move into a new directory!

How to Add an Email Alias using script

While Mac OS X does include a standard Unix mail transport system built around the venerable and remarkably complex sendmail system, it doesn't enable this system by default. To do that requires a number of complex steps, a task more complex than we can discuss in this book. If you do have sendmail running on your Mac OS X box, however, you'll doubtless want to be able to add mail aliases in a simple manner. But mail aliases aren't in /etc/aliases anymore; they're now part of the NetInfo system. This script offers an easy work-around.

Setting up sendmail A variety of sites offer instructions on how to set up sendmail on your Mac OS X system, and some simple freeware applications are even available to set it up automatically. If you want to do it yourself, go to O'Reilly's MacDevCenter at http://www.macdevcenter.com/ and search for "sendmail," or take the easy way out and go to Version Tracker at http://www.versiontracker.com/ and, again, search for "sendmail" to find a variety of freeware configuration utilities. Make sure the solution you try is for your exact version of Mac OS X.


The Code
#!/bin/sh

# addmacalias - Adds a new alias to the email alias database on Mac OS X.
# This presumes that you've enabled sendmail, which can be kind of
# tricky. Go to http://www.macdevcenter.com/ and search for "sendmail"
# for some good reference works.

showaliases="nidump aliases ."

if [ "$(/usr/bin/whoami)" != "root" ] ; then
echo "$(basename $0): You must be root to run this command." >&2
exit 1
fi

if [ $# -eq 0 ] ; then
echo -n "Alias to create: "
read alias
else
alias=$1
fi

# Now let's check to see if that alias already exists...
if $showaliases | grep "${alias}:" >/dev/null 2>&1 ; then
echo "$0: mail alias $alias already exists" >&2
exit 1
fi

# Looks good. let's get the RHS and inject it into NetInfo

echo -n "pointing to: "
read rhs # the right-hand side of the alias

niutil -create . /aliases/$alias
niutil -createprop . /aliases/$alias name $alias
niutil -createprop . /aliases/$alias members "$rhs"

echo "Alias $alias created without incident."

exit 0

How It Works
If you've studied Script #94, Adding a User to a Mac OS X System, you should immediately see all the similarities between that script and this one, including the test for root user and the invocations to niutil with the flags -create and -createprop.

The most interesting snippet in this script is the test to see if the alias already exists:

if $showaliases | grep "${alias}:" >/dev/null 2>&1 ; then
echo "$0: mail alias $alias already exists" >&2
exit 1
fi

It's a good example of how to properly use the result of a command as a test while discarding any output, either to stdout or stderr. The notation >/dev/null discards stdout, of course, and then the odd notation 2>&1 causes output device #2, stderr, to be mapped to output device #1, stdout, also effectively routing stderr to /dev/null.

Running the Script
This script is fairly flexible: You can specify the alias you'd like to create on the command line, or it'll prompt for the alias if you've forgotten. Otherwise, it prompts for needed fields and has no command flags.

The Results
$ sudo addmacalias
Alias to create: gareth
pointing to: gareth@hotmail.com
Alias gareth created without incident.

Hacking the Script
It would be quite easy to add an -l flag or something similar to addmacalias to produce a listing of all current mail aliases, and that would significantly improve the utility of this simple script.

How to Add a User to a Mac OS X System using script

you saw the basic steps involved in adding a new user to a typical Unix or Linux system. The Mac OS X version is fundamentally quite similar. In essence, you prompt for an account name and login shell, append the appropriate information to the /etc/passwd and /etc/shadow files, create the new user's home directory, and set an initial password of some sort. With Mac OS X it's not quite this simple, because appending information to /etc/passwd will not create a new Aqua account. Instead, the information must be injected into the NetInfo system using the niutil command.

The Code
#!/bin/sh

# addmacuser - Adds a new user to the system, including building the
# home directory, copying in default config data, etc.
# You can choose to have every user in his or her own group (which requires
# a few tweaks) or use the default behavior of having everyone put
# into the same group. Tweak dgroup and dgid to match your own config.

dgroup="guest"; dgid=31 # default group and groupid
hmdir="/Users"
shell="uninitialized"

if [ "$(/usr/bin/whoami)" != "root" ] ; then
echo "$(basename $0): You must be root to run this command." >&2
exit 1
fi

echo "Add new user account to $(hostname)"
echo -n "login: " ; read login

if nireport . /users name | sed 's/[^[:alnum:]]//g' | grep "^$login$" ; then
echo "$0: You already have an account with name $login" >&2
exit 1
fi

uid1="$(nireport . /users uid | sort -n | tail -1)"
uid="$(( $uid1 + 1 ))"

homedir=$hmdir/$login

echo -n "full name: " ; read fullname

until [ -z "$shell" -o -x "$shell" ] ; do
echo -n "shell: " ; read shell
done

echo "Setting up account $login for $fullname..."
echo "uid=$uid gid=$dgid shell=$shell home=$homedir"

niutil -create . /users/$login
niutil -createprop . /users/$login passwd
niutil -createprop . /users/$login uid $uid
niutil -createprop . /users/$login gid $dgid
niutil -createprop . /users/$login realname "$fullname"
niutil -createprop . /users/$login shell $shell
niutil -createprop . /users/$login home $homedir

niutil -createprop . /users/$login _shadow_passwd ""

# adding them to the $dgroup group
niutil -appendprop . /groups/$dgroup users $login

if ! mkdir -m 755 $homedir ; then
echo "$0: Failed making home directory $homedir" >&2
echo "(created account in NetInfo database, though. Continue by hand)" >&2
exit 1
fi

if [ -d /etc/skel ] ; then
ditto /etc/skel/.[a-zA-Z]* $homedir
else
ditto "/System/Library/User Template/English.lproj" $homedir
fi
chown -R ${login}:$dgroup $homedir

echo "Please enter an initial password for $login:"
passwd $login

echo "Done. Account set up and ready to use."
exit 0

How It Works
This script checks to ensure that it's being run by root (a non-root user would generate permission errors with each call to niutil and the mkdir calls, and so on) and then uses the following test to check whether the specified account name is already present in the system:

nireport . /users name | sed 's/[^[:alnum:]]//g' | grep "^$login$"

You've already seen in Script #93 that nireport is the easy way to interface with the NetInfo system, so it should be straightforward that this call generates a list of account names. It uses sed to strip all spaces and tabs and then uses grep to search for the specified login name, left rooted (^ is the beginning of the line) and right rooted ($ is the end of the line). If this test succeeds, the script outputs an error and quits.

The script also uses nireport to extract the highest user ID value in the NetInfo database and then increments it by 1 to generate the new account ID value:

uid1="$(nireport . /users uid | sort -n | tail -1)"
uid="$(( $uid1 + 1 ))"

Notice the use of the -n flag with sort to ensure that sort organizes its results from lowest to highest (you can reverse it with -nr instead, but that wouldn't work in this context), and then the use of tail -1 to pull off just the highest uid on the list.

The user is then prompted to enter a login shell over and over until either it's matched to an executable program or it's ascertained to be an empty string (empty strings default to /bin/sh as the login shell):

until [ -z "$shell" -o -x "$shell" ] ; do
echo -n "shell: " ; read shell
done


And finally we're ready to create the actual account in the NetInfo database with niutil. The first line creates an entry for the account in NetInfo, using -create, and the subsequent account attributes are added with -createprop. Notice that a special _shadow_passwd field is created, though its value is left as null. This is actually a placeholder for the future: NetInfo doesn't store the encrypted password in a secret place. Yet.

Instead of using cp -R to install user files and directories into the new account, the script uses a Mac OS X-specific utility called ditto. The ditto command ensures that any files that might have special resource forks (an Aqua-ism) are copied intact.

Finally, to force the password to be set, the script simply calls passwd with the special notation that only the root user can utilize: passwd account, which sets the password for the specified account.

Running the Script
This script prompts for input, so no command flags or command-line arguments are necessary.

The Results
$ addmacuser
addmacuser: You must be root to run this command.

Like any administrative command, this one must be run as root rather than as a regular user. This is easily solved with the sudo command:

$ sudo addmacuser
Add new user account to TheBox.local.
login: gareth
full name: Gareth Taylor
shell: /bin/bash
Setting up account gareth for Gareth Taylor...
uid=508 gid=31 shell=/bin/bash home=/Users/gareth
Please enter an initial password for gareth:
Changing password for gareth.
New password:
Retype new password:
Done. Account set up and ready to use.

That's all there is to it. Figure 11-1 shows the login window with account gareth as one of the choices.


Figure 11-1: Login window with Gareth's account included
Hacking the Script
Probably the greatest adjustment that might be required for this script is to change the group membership model. Currently the script is built to add all new users to the guest group, with the group ID specified as dgid at the beginning of the script. While many installations might work fine with this setup, other Mac OS X sites emulate the Linux trick of having every user in his or her own group. To accomplish that, you'd want to add a block of new code that auto-generates a group ID one value higher than the largest group ID currently in the NetInfo database and then instantiates the new group using the niutils command:

niutil -create . /groups/$login


Another nice hack might be to automatically email new users a welcome message, so that when they first open up their mailer there are some basic instructions on how to work with the system, what the default printer is, and any usage and network access policies.

How to tweak linux for Displaying a File with Line Numbers using shell script

There are a lot of ways to add line numbers to a displayed file, many of which are quite short. Here's a solution using awk:

awk '{ print NR": "$0 }' < inputfile

On some Unix implementations, the cat command has an -n flag, and on others, the more (or less, or pg) pager has a flag for specifying that each line of output should be numbered. But on some Unixes, none of these will work, in which case the simple script given here can do the job.

The Code
#!/bin/sh

# numberlines - A simple alternative to cat -n, etc.

for filename
do
linecount="1"
while read line
do
echo "${linecount}: $line"
linecount="$(($linecount + 1))"
done < $filename
done
exit 0

Running the Script
You can feed as many filenames as you want to this script, but you can't feed it input via a pipe, though that wouldn't be too hard to fix, if needed.

The Results
$ numberlines text.snippet.txt
1: Perhaps one of the most valuable uses of shell scripts is to fix
2: your particular flavor of Unix and make it more like other flavors,
3: to bring your commands into conformance or to increase consistency
4: across different systems. The outsider view of Unix suggests a
5: nice, uniform command-line experience, helped along by the existence
6: of and compliance with the POSIX standards for Unix. But anyone who's
7: ever touched more than one computer knows how much they can vary
8: within these broad parameters.

Hacking the Script
Once you have a file with numbered lines, you can also reverse the order of all the lines in the file:

cat -n filename | sort -rn | cut -c8-

This does the trick on systems supporting the -n flag to cut, for example. Where might this be useful? One obvious situation is when displaying a log file in most-recent-to-least-recent order.

How to Display the Contents of Directories using shell Script

While the ls command is a cornerstone of working with the Unix command line, there's one element of the command that's always seemed pointless to me: indicating the size of a directory. When a directory is listed, the program either lists the directory's contents file by file or shows the number of 1,024-byte blocks required for the directory data. A typical entry in an ls -l output might be

drwxrwxr-x 2 taylor taylor 4096 Oct 28 19:07 bin

But that's really not very useful, because what I want to know is how many files are in the specified directory. That's what this script accomplishes, generating a nice multicolumn listing of files and directories that shows file size with file entries and the number of files with directory entries.

The Code
#!/bin/sh

# formatdir - Outputs a directory listing in a friendly and useful format.

gmk()
{
# Given input in Kb, output in Kb, Mb, or Gb for best output format
if [ $1 -ge 1000000 ] ; then
echo "$(scriptbc -p 2 $1 / 1000000)Gb"
elif [ $1 -ge 1000 ] ; then
echo "$(scriptbc -p 2 $1 / 1000)Mb"
else
echo "${1}Kb"
fi
}

if [ $# -gt 1 ] ; then
echo "Usage: $0 [dirname]" >&2; exit 1
elif [ $# -eq 1 ] ; then
cd "$@"
fi

for file in *
do
if [ -d "$file" ] ; then
size=$(ls "$file" | wc -l | sed 's/[^[:digit:]]//g')
if [ $size -eq 1 ] ; then
echo "$file ($size entry)|"
else
echo "$file ($size entries)|"
fi
else
size="$(ls -sk "$file" | awk '{print $1}')"
echo "$file ($(gmk $size))|"
fi
done | \
sed 's/ /^^^/g' | \
xargs -n 2 | \
sed 's/\^\^\^/ /g' | \
awk -F\| '{ printf "%-39s %-39s\n", $1, $2 }'

exit 0

How It Works
One of the most interesting parts of this script is the gmk function, which, given a number in kilobytes, outputs that value in kilobytes, megabytes, or gigabytes, depending on which unit is most appropriate. Instead of having the size of a very large file shown as 2083364KB, for example, this function will instead show a size of 2.08GB. Note that gmk is called with the $() notation in the following line:

echo "$file ($(gmk $size))|"

Because the arguments within the $() sequence are given to a subshell of the running script shell, subshells automatically inherit any functions defined in the running shell.

Near the top of the script, there is also a shortcut that allows users to specify a directory other than the current directory and then changes the current working directory of the running shell script to the desired location, using cd. This follows the mantra of good shell script programming, of course: Where there's a shortcut, there's a better way.

The main logic of this script involves organizing the output into two neat, aligned columns. You can't make a break at spaces in the output stream, because files and directories can have spaces within their names. To get around this problem, the script first replaces each space with a sequence of three carets (^^^). Then it uses the xargs command to merge paired lines so that every two lines become one line separated by a space. Finally, it uses the awk command (rather than paste, which would just intersperse a tab, which rarely, if ever, works out properly because paste doesn't take into account variation in entry width) to output columns in the proper alignment.

Notice how the number of (nonhidden) entries in a directory is easily calculated, with a quick sed invocation cleaning up the output of the wc command:

size=$(ls "$file" | wc -l | sed 's/[^[:digit:]]//g')


Running the Script
For a listing of the current directory, invoke the command without arguments. For information about the contents of a particular directory, specify a directory name as the sole command argument.

The Results
$ formatdir ~
Applications (0 entries) Classes (4Kb)
DEMO (5 entries) Desktop (8 entries)
Documents (38 entries) Incomplete (9 entries)
IntermediateHTML (3 entries) Library (38 entries)
Movies (1 entry) Music (1 entry)
NetInfo (9 entries) Pictures (38 entries)
Public (1 entry) RedHat 7.2 (2.08Gb)
Shared (4 entries) Synchronize! Volume ID (4Kb)
X Desktop (4Kb) automatic-updates.txt (4Kb)
bin (31 entries) cal-liability.tar.gz (104Kb)
cbhma.tar.gz (376Kb) errata (2 entries)
fire aliases (4Kb) games (3 entries)
junk (4Kb) leftside navbar (39 entries)
mail (2 entries) perinatal.org (0 entries)
scripts.old (46 entries) test.sh (4Kb)
testfeatures.sh (4Kb) topcheck (3 entries)
tweakmktargs.c (4Kb) websites.tar.gz (18.85Mb)

Hacking the Script
The GNU version of ls has an -h flag that offers similar functionality. If you have that version of ls available, adding that flag and removing the call to gmk will speed up this script.

The other issue worth considering with this script is whether you happen to have a user who likes to use sequences of three carets in filenames, which could cause some confusion in the output. This naming convention is pretty unlikely, however. A 116,696-file Linux install that I spot-tested didn't have even a single caret within any of its filenames. However, if you really are concerned, you could address this potential pitfall by translating spaces into another sequence of characters that's even less likely to occur in user filenames.

How to Log File Removals using Shell Script

The basic idea of wrappers is that they live between an actual Unix command and the user, offering the user different and useful functionality not available with the actual command alone. In the case of this script, file deletions using the rm command will actually be logged in a separate log file without notifying the user.

The Code
#!/bin/sh
# logrm - Logs all file deletion requests unless the -s flag is used.

removelog="/var/log/remove.log"

if [ $# -eq 0 ] ; then
echo "Usage: $0 [-s] list of files or directories" >&2
exit 1
fi

if [ "$1" = "-s" ] ; then
# silent operation requested ... don't log
shift
else
echo "$(date): ${USER}: $@" >> $removelog
fi

/bin/rm "$@"

exit 0

Running the Script
Rather than give this script a name like logrm, a typical way to install a wrapper program is to rename the underlying program and then install the wrapper using the underlying program's old name. If you choose this route, make sure that the wrapper invokes the newly renamed program, not itself. For example, if you rename /bin/rm to /bin/rm.old and name this script /bin/rm, the last few lines of the script will need to be changed so that it invokes /bin/rm.old, not itself!

You can also use an alias to have this script wrap a standard call to rm:

alias rm=logrm

In either case, you will, of course, need write and execute access to /var/log, which might not be the default configuration on your particular Unix or Mac OS X system.

The Results
Let's create a few files to delete, delete them, and then examine the remove log:

$ touch unused.file ciao.c /tmp/junkit
$ logrm unused.file /tmp/junkit
$ logrm ciao.c
$ cat /var/log/remove.log
Thu Jul 3 11:32:05 MDT 2003: susan: /tmp/central.log
Fri Jul 4 14:25:11 MDT 2003: taylor: unused.file /tmp/junkit
Fri Jul 4 14:25:14 MDT 2003: taylor: ciao.c

Aha! Notice that on the previous day user susan deleted the file /tmp/central.log.

Hacking the Script
There's a potential log file ownership permission problem here too. Either the remove.log file is writable by all, in which case a user could clear its contents out with a command like cat /dev/null > /var/log/remove.log, or it isn't writable by all, in which case the script can't log the events. You could use a setuid permission so that the script runs with the same permissions as the log file, but there are two problems with this. First, it's a really bad idea! Never run shell scripts under setuid! Second, if that's not enough of a reason, you could get into a situation where the users have permission to delete their files but the script doesn't, and because the effective uid set with the setuid would be inherited by the rm command itself, things would break and there would be great confusion when users couldn't remove their own files, even when they check and see that they own the files in question.

Two other possible solutions to this problem are worth mentioning. First, if you have an ext2 or ext3 file system (probably Linux), you can use the chattr command to set a specific append-only file permission on the log file and then leave it writable to all without any danger. Second, you can write the log messages to syslog, using the helpful logger command. To log the rm commands with logger is straightforward:

logger -t logrm "${USER:-LOGNAME}: $*"

This adds an entry to the syslog data stream (untouchable by regular users) that is tagged with logrm, the username, and the command specified.

Syslog nuances to watch for If you opt for this approach, you'll want to check syslogd(8) to ensure that your configuration doesn't discard user.notice priority log events (it's almost always specified in the /etc/ syslogd.conf file).