AWS Reporting

The following is an introduction to the AWS reporting package. I've also tried to give an overview of what's going on with AWS as a whole; Martha's and Larry's pages document some of this stuff in admirable detail.
Note that most of the hyperlinks below point to copies of the files, not the actual "live" files. This means that if the originals get modified, the copies may or may not be out of date. I'll try to keep them up, but your best bet for absolute accuracy is to check these things where they live.

The Flow
The Components
Specific Tasks

To take a station up or down
If reports are broken
Ugly things happen around New Years Day...

The Flow

The ThinkPad in Bill's office runs a batch file called AWS. (TODO: remove pipe chars from comments in aws.bat!) The batch file uses a time-polling system to decide when to call the stations and do other things. When the correct time rolls around, it calls CALLSTN.BAT.

CALLSTN.BAT sets an environment variable to the abbreviations of the station names (ARK, CHE, etc.). This makes the abbrevs available, and also serves to count the stations. (So if the number of stations changes, you have to change this SET statement).
The data files get copied to //Bob//work/aws/xfer/dat/... semaphore file is created in //Bob//work/aws/xfer/ready... AWSWAIT.PL sees it...and data gets entered into the files on Bob, winding up in //Bob/work/bbs/data/(STN)(YR)(RECID).TXT.

A C-shell script called ALLREPORTS.CSH gets called from AWSWAIT.PL. This script uses the GETDATA executable to extract the appropriate data for the days we're interested in. The text files output by the various calls to GETDATA then get massaged into nice-looking reports by two C programs called CLIMREPORTand SOILREPORT. Four reports wind up being generated by ALLREPORTS.CSH: clim.rpt, soil.rpt, 7dayclim.rpt, and 7daysoil.rpt. ALLREPORTS.CSH uses a database of stations to determine which stations get reported, what sensors they have, their locations, and what kinds of data they produce.

The Components

There are a variety of software and data components which cooperate to make all this happen. Specifically:

Programs and Scripts

//ThinkPad/aws/netmngr/aws.bat controls downloading from stations using Campbell Scientific's TelCom EXE; it also copies the data onto a shared NFS drive for further processing by the UNIX boxes, and uses PKZIP to create archives of the raw data files on the hard disk and on the shared NFS drive

//ThinkPad//aws/netmngr/callstn.bat actually does the calling, using Telcom.
//bob/bill/products/sh/aws.sh: compiles the raw data files into the text files

//bob/bill/products/pl/awswait.pl: master Perl script to run AWS stuff on the UNIX side

//bob/~wayne/reports/: contains the source code for the controlling shell script allreports.csh, the exectuables climreport and soilreport, and their utility program rd_stations.

//ThinkPad/aws/????: directory where TelCom software lives

chkstnyy: program that checks the dat file name against the station list, sanity-checks the year and day (note that the year check is NOT Y2K compliant). It then goes into the file itself and checks that each line roughly agrees with the AWS format (RECID,STN,YYDDD,HHMM). Next it checks that the line's year stamp agrees with the name of the file, and does some range-checking on the day-of-year.
only2400: filter data lines so that only 2400-timestamped ones get through, and log an error message for any others.

calcet: calculate reference ET and clear ratio in an AWS line, add them (and the edit code) to the end of the line. Pass other lines through unchanged. note: this means you don't have to splitrec first!

noflds: checks the number of comma-separated fields on each stdin line against the number passed in on the command line, and filters out the line (logging an error message) if not. Handles the "edit field at EOL.

fix401: The 401 record has variant definitions. This program pads the records with only one rain bucket with a missing-value field.

Data

//bob/work/aws/sysdata/stnlist.new

//bob/work/bbs/data/newfld.dat

Specific Tasks

If a station goes up or down

To change a station's status, you have to whack on several different files. Note that the hyperlinks here will show you the file you need to edit -- the copy of the file, that is!

AWS.BAT on the ThinkPad: Remove the station abbreviation from the list.
STNLIST.NEW: cd to /work/aws/sysdata. su to "aws", if you haven't already. Check the station list out ("co -l stnlist.new"). Edit the file, change "up" to "down" or vice versa, save it out, and check the file back in ("ci -u stnlist.new").
Special case for particular stations: if you take down ANTigo, CHEtek, CRAnmoor, or HANcock, go into the "spud-mailing" perl script ($bp/pl/emailspuds.pl) and modify the list of reports there.

If the reports don't appear to be running

If you notice that the reports haven't been updating for a day or two, the most likely culprit is the ThinkPad.

Check to see that it's not locked up. Try pressing the Fn/NumLock key combination and see if the NumLock light goes on and off -- if not, the system is completely toast and needs to be reset -- maybe power-cycled.
Get the screen to light up by pressing the space bar, or by pressing the button on the right side of the unit near the display hinge.
What's the screen showing? If you see a bunch of lines giving the time, and another one appears each minute, then at least the AWS script is running. To exit the AWS script, press the Ctrl and Brk keys, and wait for awhile...awhile...awhile...heck, if you get too impatient just hit Ctrl-Alt-Del and reboot the thing; it will come up with a DOS prompt.
If a PKZIP operation failed, the hard disk might be full; do a DIR to see how much space is left. The disk might also be corrupt; use CHKDSK to look for errors, and CHKDSK /f to repair them. Note that the corruption/repair process can remove files, maybe important files, so it's a good idea to know where the backups are. There is a PKZIP image of the ThinkPad's hard disk on Rick's machine, LEELA, on the D: drive ("Projects & Data") under \AWS\laptop.

Special Christmas Bonus Ugly Kludge Special

Because of a problem with how getdata handles dates, the reporting stuff goes haywire for the week that spans the new year. Some details on how to fix it are in ALLREPORTS.CSH, but I will try to document it better here too. Real Soon Now.