Awk as a Templating Language

Preface

This is an article I wrote for work but I thought that it may be useful to others as well. It describes how to use awk as a simple templating language to run multiple command in a linux/unix environment. It talk specifically about a command we use called btq_submit which submits reports to our report queuing system but you could applied the technique to any comandline command.

This is a technique to use AWK to create an input and template to run reports. This method has a lot of over head so it will most likely only be useful where there is a report ran for many different sets of vars.

Overview

The command has 2 parts. The first part is the input. The input is what provides the information to the template. The template is the second part. It takes the input and writes out the btq_submit block.

How AWK Works

What AWK does is that it takes input and then brakes that input in to records and fields. It then preforms actions on those records and field and produces output. If you look at our input section each line is one record. Each field in the record is separated by a :

example

• Is one record
  • “\n” tells awk that this is the end of the record
• Has 4 fields
  • field 1 is “run”
  • field 2 is “FURN”
  • field 3 is “310″
  • field 4 is “574Z”

Now if we want AWK to do something when it sees this record we need to make a block for it in the AWK command.

example

The first part (/^run/) means: if the record start with run then do the following. The part inside the {} is the block of code to run. Here we are setting 3 vars from the feilds in the record. $[field number] is how you access the text in a field. $2 is the second field $3 is the third field … The last line is a print command. The sections in “” will be printed as text. Var names not in quotes will be substituted with there value.

The result returned by AWK for the above example is: dept is FURN, start is 310, end is 574Z

Parts of the Command

echo”[input]“|awk ‘[awk program]‘|sh

You must first echo your input and pipe it into the AWK command. You must then pipe the output of the AWK command to the shell if you want it to run the reports. You must have the “|sh” on the end if you want the script outputted by the AWK command to run. If you leave it off it will just output to the screen. This can be useful for debugging.

BEGIN block

At the start of the awk program there is a special block names BEGIN. The BEGIN block will execute before anything else in the awk program. In the begin block you have to set the separators that awk uses to tell the records and fields apart in the input. If you do not set these awk will default to RS=”\n” and FS=” “. Not normally what you want.

RS is the record separator

FS is the field separators

Setting Up the Vars

There are 2 ways to setup vars in this style of script. There is are persistent vars and there are run time vars.

Persistent

If you have one var that will be the same for several reports but not all reports you can use this method. If you look at the example the parts dealing with persistent vars are in green . In the input section they are written as [name]:[value1]:[value2] …  you can have as many values after the name as you want.

In the AWK section of the script you can see the sections that set the persistent var. the way these are written is

The part in green is how AWK knows what to do with the line from the input. AWK reads each line and then looks for any blocks to apply to that line. What the /^name/ means to AWK is “if the record starts with [name] then do the following”.

example
If line starts with “market” then set the var market = to the second value in that line.

Once you set a var in AWK it will retain that value until it is reset or the AWK program exits. So vars you set outside the run block may be used for several reports.

At run time

If there vars that will change with each report you can include then in the run line. If you do not want to place them in their own vars you can use the $[field] notation.

The run Line

This is where AWK will output the btq_submit block. This is done using the print command. There will be 1 run line for each report you want the script to run.

example

If this line is ran it will produce an a result like the following.

Making Switch Vars

Some times you may have a report parameter that is only different once or twice in a large number of reports. It would be a lot of extra work to state this parameter for every report just so it can be different once or twice. In these cases you can use this technique to make a switch var. A switch var is different from a normal var in that it has a default value. If you set a switch var the value you set it for will be used by the next report ran. The switch will then be set back to its default value and following reports will use the default value.

In the example above the lines for the switch var are in red.

• As far as the input part of the template you use the switch var the same way as you would use a normal var.
  • I like to name them starting with a capital S to show that it is a switch var and not a regular var.
• In the BEGIN block you need to set the switch var to its default value.
• Make a set block for the switch block in the same way as you would for a normal var.
• it is used in the print btq_submit area the same as a normal var.
• at the end of the run block set the switch var back to its default

You can have more then one switch var in a script, but you will need to have all of these parts for each one.

For loops

It is also possible to add looping to awk script. see the example below

The output will look like this.

What is important to notice here is that each run line will submit a report for each of the vars that the for_ var is set for. It is also important to notice that the for_ var remands set after each run line. So multiple runs can be made with out reseting the for_ var.