Mechanical Web Authoring - using m4 to write HTML and Perl.

Contents:

This page last updated on Mon Mar 25 10:28:28 EST 2002
$Revision: 1.10 $

1. Some limitations of HTML

It's amazing how easy it is to write simple HTML pages - and the availability of WYSIWYG HTML editors like NETSCAPE COMPOSER lulls one into a mood of "don't worry, be happy". However, managing multiple, interrelated pages of HTML rapidly gets very, very difficult. I recently had a slightly complex set of pages to put together and it started me thinking - "there has to be an easier way".

I immediately turned to the WWW and looked up all sorts of tools - but quite honestly I was rather disappointed. Mostly, they were what I would call Typing Aids - instead of having to remember arcane incantations like <a href="link">text</a>, you are given a button or a magic keychord like ALT-CTRL-j which remembers the syntax and does all that nasty typing for you.

Linux to the rescue! HTML is built as ordinary text files and therefore the normal Linux text management tools can be used. This includes the revision control tools such as RCS and the text manipulation tools like awk, perl, etc. These offer significant help in version control and managing development by multiple users as well as in automating the process of extracting from a database and displaying the results (the classic "grep |sort |awk" pipeline).

The use of these tools with HTML is documented elsewhere, e.g. see Jim Weinrich's article in Linux Journal Issue 36, April 1997, "Using Perl to Check Web Links" which I'd highly recommend as yet another way to really flex those Linux muscles when writing HTML.

What I will cover here is a little work I've done recently with using m4 in maintaining HTML. The ideas can probably be extended to the more general SGML case very easily.

I decided to use m4 after looking at various other pre-processors including cpp, the C front-end. While cpp is perhaps a little too C-specific to be very useful with HTML, m4 is a very generic and clean macro expansion program - and it's available under most Unices including Linux.

Instead of editing *.html files, I create *.m4 files with my favourite text editor. These look something like this: 

m4_include(stdlib.m4)
_HEADER(`This is my header')
<P>This is some plain text<P>
_HEAD1(`This is a main heading')
<P>This is some more plain text<P>
_TRAILER

The format is simple - just HTML code but you can now include files and add macros rather like in C. I use a convention that my new macros are in capitals and start with "_" to make them stand out from HTML language and to avoid name-space collisions.

The m4 file is then processed as follows to create an .html file e.g. 

m4 -P <file.m4 >file.html
This is especially easy if you create a "makefile" to automate this in the usual way. Something like: 
.SUFFIXES: .m4 .html
.m4.html:
	m4 -P $*.m4 >$*.html
default: index.html
*.html: stdlib.m4
all: default PROJECT1 PROJECT2
PROJECT1:
	(cd project2; make all)
PROJECT2:
	(cd project2; make all)
The most useful commands in m4 include the following which are very similar to the cpp equivalents (shown in brackets):
m4_include:
includes a common file into your HTML (#include)
m4_define:
defines an m4 variable (#define)
m4_ifdef, m4_ifelse:
conditionals (#ifdef, #if)
Some other commands which are useful are:
m4_changecom:
change the m4 comment character (normally #)
m4_debugmode:
control error disgnostics
m4_traceon/off:
turn tracing on and off
m4_dnl:
comment
m4_incr, m4_decr:
simple arithmetic
m4_eval:
more general arithmetic
m4_esyscmd:
execute a Linux command and use the output
m4_divert(i):
This is a little complicated, so skip on first reading. It is a way of storing text for output at the end of normal processing - it will come in useful later, when we get to automatic numbering of headings. It sends output from m4 to a temporary (internal) file number i. At the end of processing, any text which was diverted is then output, in the order of the file number i. File number -1 is the bit bucket and can be used to comment out chunks. File number 0 is the normal output stream. Thus, for example, you can m4_divert text to file 1 and it will only be output at the end.
Contents

2. Examples of m4 macros

2.1 Typing and mnemonic aids

I don't always depend on WYSIWYG editing (having been brought up on troff) but all the same I'm not averse to taking the easy way out where it's available. There is a choice (and maybe it's a fine line) to be made between: 
<BLOCKQUOTE><PRE><CODE>Some code you want to display.
</CODE></PRE></BLOCKQUOTE>
and: 
_CODE(Some code you want to display.)
In this case, you would define _CODE like this: 
m4_define(`_CODE',
	 <BLOCKQUOTE><PRE><CODE>$1</CODE></PRE></BLOCKQUOTE>)
Which version you prefer is a matter of taste and convenience although the m4 macro certainly saves some typing and ensures that HTML codes are not interleaved. Another example I like to use (I can never remember the syntax for links) is: 
m4_define(`_LINK', <a href="$1">$2</a>)
Then, <a href="URL_TO_SOMEWHERE">Click here to get to SOMEWHERE </a> becomes: _LINK(`URL_TO_SOMEWHERE', `Click here to get to SOMEWHERE')

Contents

2.2 Creating new text styles

Styles built into HTML include things like <EM> for emphasis and <CITE> for citations. With m4 you can define your own, new styles like this: 
m4_define(`_MYQUOTE',
	<BLOCKQUOTE><EM>$1</EM></BLOCKQUOTE>)
If, later, you decide you prefer <STRONG> instead of <EM> it is a simple matter to change the definition and then every _MYQUOTE paragraph falls into line with a quick make.

The classic guides to good HTML writing say things like "It is strongly recommended that you employ the logical styles such as <EM>...</EM> rather than the physical styles such as <I>...</I> in your documents." Curiously, the WYSIWYG editors for HTML generate purely physical styles. Using these m4 styles may be a good way to keep on using logical styles.

Contents

2.3 Sharing HTML elements across several page

In many "nests" of HTML pages, each page shares elements such as a button bar like this:
[Home] [Next] [Prev] [Index]
This is fairly easy to create in each page - the trouble is that if you make a change in the "standard" button-bar then you then have the tedious job of finding each occurrence of it in every file and then manually make the changes.

With m4 we can more easily do this by putting the shared elements into an m4_include statement, just like C.

While I'm at it, I might as well also automate the naming of pages, perhaps by putting the following into an include file, say "button_bar.m4": 

m4_define(`_BUTTON_BAR', 
	<a href="homepage.html">[Home]</a>
	<a href="$1">[Next]</a>
	<a href="$2">[Prev]</a>
	<a href="indexpage.html">[Index]</a>)
and then in the document itself: 
m4_include button_bar.m4
_BUTTON_BAR(`page_after_this.html', 
	`page_before_this.html')
The $1 and $2 parameters in the macro definition are replaced by the strings in the macro call.

Contents

2.4 Links that change for every page

Links are often the same on every page except for one small change - you want the link to the current page to be disabled or invisible. Here's an easy way to do it with m4.

Simply define this in each page: 

m4_define(`_PAGE_1',1)
m4_include(links.m4)

Then, in the file links.m4: 

[ m4_ifdef(`_PAGE_1',`Page 1',_LINK(page1.html,`Page 1')) ]
[ m4_ifdef(`_PAGE_2',`Page 2',_LINK(page2.html,`Page 2')) ]
[ m4_ifdef(`_PAGE_3',`Page 3',_LINK(page3.html,`Page 3')) ]
[ m4_ifdef(`_PAGE_4',`Page 4',_LINK(page4.html,`Page 4')) ]

In "Page 1" you would get:

[ Page 1 ] [ Page 2 ] [ Page 3 ] [ Page 4 ]

which is, hopefully, what you'd expect. It's nice that the complexity is pushed into the included file links.m4 leaving the calling pages relatively uncluttered.

Contents

2.5 Managing HTML elements that often change

It is very troublesome to have items change in multiple HTML pages. For example, if your email address changes then you will need to change all references to the new address. Instead, with m4 you can do something like this in your stdlib.m4 file: 
m4_define(`_EMAIL_ADDRESS', `MyName@foo.bar.com')
and then just put _EMAIL_ADDRESS in your m4 files.

A more substantial example comes from building strings up with multiple components, any of which may change as the page is developed. If, like me, you develop on one machine, test out the page and then upload to another machine with a totally different address then you could use the m4_ifdef command in your stdlib.m4 file (just like the #ifdef command in cpp): 

m4_define(`_LOCAL')
.
.
m4_define(`_HOMEPAGE', 
	m4_ifdef(`_LOCAL', `//127.0.0.1/~YourAccount', 
		`http://ISP.com/~YourAccount'))

m4_define(`_PLUG', `<A REF="http://www.ssc.com/linux/">
	<IMG SRC="_HOMEPAGE/gif/powered.gif" 
	ALT="[Linux Information]"> </A>')
Note the careful use of quotes to prevent the variable _LOCAL from being expanded. _HOMEPAGE takes on different values according to whether the variable _LOCAL is defined or not. This can then ripple through the entire project as you make the pages.

In this example, _PLUG is a macro to advertise Linux. When you are testing your pages, you use the local version of _HOMEPAGE. When you are ready to upload, you can remove or comment out the _LOCAL definition like this: 

m4_dnl m4_define(`_LOCAL')
... and then re-make.

Contents

2.6 Automatic date stamping

For simple, automatic datestamping of HTML pages I use the m4_esyscmd command to maintain an automatic timestamp on every page: 
This page was updated on m4_esyscmd(date)
which produces:

This page was last updated on Fri May 9 10:35:03 HKT 1997

Of course, you could also use the date, revision and other facilities of revision control systems like RCS or SCCS, e.g. $Date$.

You can also use any system command, not just date. For example you can count the number of elements in a flat file database like this: 

There are m4_esyscmd(wc -l filename.txt) items in the database today
This might be dynamic enough for your purposes and avoids the overhead of a perl invocation every time the otherwise static page is downloaded.

Contents

2.7 Automatic numbering

m4 has a simple arithmetic facility with two operators m4_incr and m4_decr which act as you might expect - this can be used to create automatic numbering, perhaps for headings, e.g.: 
m4_define(_CARDINAL,0)

m4_define(_H, `m4_define(`_CARDINAL',
	m4_incr(_CARDINAL))<H2>_CARDINAL.0 $1</H2>')

_H(First Heading)
_H(Second Heading)

This produces: 

<H2>1.0 First Heading</H2>
<H2>2.0 Second Heading</H2>

Contents

2.8 Generating Tables of Contents

Automatic numbering naturally leads us to this topic. Even if you don't want to understand the innards of how m4 does this, you can use the feature in my stdlib.m4 for an accurate, fast and simple to use facility.

m4 allows you to define commonly repeated phrases and use them consistently - this appeals to me enormously as I hate repetition both because I am lazy and because I make mistakes. In HTML, a TOC involves repeating the heading title in the table of contents and then in the text itself. This is tedious and error-prone especially when you change the titles. There are specialised tools for generating tables of contents from HTML pages but the simple facility provided by m4 is irresistable to me.

2.8.1 Simple to understand TOC

The following example is a fairly simple-minded Table of Contents generator - the next section discusses a simple-to-use version that I use. It uses nicknames (or glossaries) for the headings instead of the headings themselves.

First, create some useful macros in stdlib.m4: 

m4_define(`_LINK_TO_LABEL', <A HREF="#$1">$1</A>)
m4_define(`_SECTION_HEADER', <H2><A NAME="$1">$1</A></H2>)
Then define all the section headings at the start of the page: 
m4_define(`_DIFFICULTIES', `The difficulties of HTML')
m4_define(`_USING_M4', `Using <EM>m4</EM>')
m4_define(`_SHARING', `Sharing HTML Elements Across Several Pages')
Then build the table: 
<UL><P>
	<LI> _LINK_TO_LABEL(_DIFFICULTIES)
	<LI> _LINK_TO_LABEL(_USING_M4)
	<LI> _LINK_TO_LABEL(_SHARING)
<UL>
Finally, write the text: 
.
.
_SECTION_HEADER(_DIFFICULTIES)
.
.
The advantages of this approach are that if you change your headings you only need to change them in one place and the table of contents is automatically regenerated; also the links are guaranteed to work.

Hopefully, that simple version was fairly easy to understand.

Contents

2.8.2 Simple to use TOC

The Table of Contents generator that I normally use is a bit more complex and will require a little more study, but is much easier to use. It not only builds the Table, but it also automatically numbers the headings on the fly - up to a maximum of 4 levels of numbering e.g. section 3.2.1.3 - although this can be easily extended. It is, however, very simple to use as follows:
  1. Where you want the table to appear, put Start_TOC
  2. at every heading use _H1(`Heading for level 1') or _H2(`Heading for level 2') as appropriate.
  3. After the very last HTML code (probably after </HTML>), put End_TOC
  4. and that's all!
The code for these macros is a little complex, so hold your breath: 
m4_define(_Start_TOC,`<UL><P>m4_divert(-1)
  m4_define(`_H1_num',0)
  m4_define(`_H2_num',0)
  m4_define(`_H3_num',0)
  m4_define(`_H4_num',0)
  m4_divert(1)')

m4_define(_H1, `m4_divert(-1)
  m4_define(`_H1_num',m4_incr(_H1_num))
  m4_define(`_H2_num',0)
  m4_define(`_H3_num',0)
  m4_define(`_H4_num',0)
  m4_define(`_TOC_label',`_H1_num. $1')
  m4_divert(0)<LI><A HREF="#_TOC_label">_TOC_label</A>
  m4_divert(1)<H2><A NAME="_TOC_label">
	_TOC_label</A></H2>')
.
.
[definitions for _H2, _H3 and _H4 are similar and are 
in the downloadable version of stdlib.m4]
.
.

m4_define(_End_TOC,`m4_divert(0)</UL><P>')

This works by using the m4_divert(1) command in _Start_TOC to send all the remaining text from the file to an (internal) temporary file - m4 just calls it "file 1".

From then on, whenever an _H1 or _H2 etc command is reached, the relevant header numbering variables are incremented (with m4_incr) and the Table of Contents entry is sent to the standard output (file 0) together with automatically generated pointers into the main text.

The diversion to file 1 is then resumed for the regular text and an automatically generated section heading with pointer target is added.

The _End_TOC statement must be placed at the end of the file. When it is reached the text which was diverted to file 1 is read back to standard output.

The net result is that the Table of Contents appears near the start of the final file, with automatically generated pointers to the correct section in the later text.

Of course, if you plan on using the m4_divert command in your own text, you will have to check that it does not clash with the Table of Contents generator.

As a final note on this feature, the table of contents generator in the downloadable version of stdlib.h also indents the sections. For the sake of a simpler explanation this extra feature is not explained here (or in the words of my Analysis Lecturer, Prof. Noble, "it is left as an excercise for the student").

Contents

2.9 Simple tables

Other than Tables of Contents, many browsers support tabular information. Here are some funky macros as a short cut to producing these tables. First, an example of their use: 
<CENTER>
_Start_Table(BORDER=5)
_Table_Hdr( ,Apples, Oranges, Lemons)
_Table_Row(England,100,250,300)
_Table_Row(France,200,500,100)
_Table_Row(Germany,500,50,90)
_Table_Row(Spain, ,23,2444)
_Table_Row(Denmark, , ,20)
_End_Table
</CENTER>
 ApplesOrangesLemons
England100250300
France200500100
Germany5005090
Spain 232444
Denmark  20

...and now the code. Note that this example utilises m4's ability to recurse: 

m4_dnl _Start_Table(Columns,TABLE parameters)
m4_dnl defaults are BORDER=1 CELLPADDING="1" CELLSPACING="1"
m4_dnl WIDTH="n" pixels or "n%" of screen width
m4_define(_Start_Table,`<TABLE $1>')

m4_define(`_Table_Hdr_Item', `<th>$1</th>
  m4_ifelse($#,1,,`_Table_Hdr_Item(m4_shift($@))')')

m4_define(`_Table_Row_Item', `<td>$1</td>
  m4_ifelse($#,1,,`_Table_Row_Item(m4_shift($@))')')

m4_define(`_Table_Hdr',`<tr>_Table_Hdr_Item($@)</tr>')
m4_define(`_Table_Row',`<tr>_Table_Row_Item($@)</tr>')

m4_define(`_End_Table',</TABLE>)

Contents

3. Using m4 with Perl

This works too - you can have common header files for your HTML files and perl CGI scripts. These might define common constants and look and feel.

e.g. if _HEADER and _DocumentRoot are defined as: 


m4_define(`_HEADER',<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML> 
<HEAD> 
  <TITLE>$1</TITLE>
  <META NAME="Author" CONTENT="Bob Hepple">
  <META NAME="Description" CONTENT="$3">
  <META NAME="Keywords" CONTENT="$4">
</HEAD>)
m4_define(`_DocumentRoot',/cars)

then we can define the following in a perl script 

m4_include(stdlib.m4)
print "_HEADER(title,params,description,keywords)";
print "_LINK(_DocumentRoot/a_file.html,a link)\n";

and in a plain HTML page: 

m4_include(stdlib.m4)
_HEADER(title,params,description,keywords)

We then get the advantages of consistency between HTML and perl scripts, easy global changes and most of the complexity moved off to the include-file.

perl's 'here document' construct is also amenable to macro expansion with m4. e.g. 


print <<Eof;
_HEADER(title,params,description,keywords)
Eof

I use this process in my own pages e.g. at The Finder which is a commercial site for classic cars in Australia. All scripts and HTML pages are generated from m4 source and result in a consistent look and feel inherited from a single include-file.

Contents

4. m4 gotchas

Unfortunately, m4 is not unremitting sweetness and light - it needs some taming and a little time spent on familiarisation will pay dividends. Definitive documentation is available (for example in emacs' info documentation system) but, without being a complete tutorial, here are a few tips based on my fiddling about with the thing.

4.1 Gotcha 1 - quotes

m4's quotation characters are the grave accent ` which starts the quote, and the acute accent ´ which ends it. It generally helps to put all arguments to macros in quotes, e.g. 
_HEAD1(`This is a heading')
The main reason for this is in case there are commas in an argument to a macro - m4 uses commas to separate macro parameters, e.g. _CODE(foo, bar) would print the foo but not the bar. _CODE(`foo, bar') works properly.

This becomes a little complicated when you nest macro calls as in the m4 source code for the examples in this paper - but that is rather an extreme case and normally you would not have to stoop to that level.

Contents

4.2 Gotcha 2 - Word swallowing

The worst problem with m4 is that some versions of it "swallow" key words that it recognises, such as "include", "format", "divert", "file", "gnu", "line", "regexp", "shift", "unix", "builtin" and "define". You can protect these words by putting them in m4 quotes, for example: 
Smart people `include' Linux in their list of computer essentials.
The trouble is, this is a royal pain to do - and you're likely to forget which words need protecting.

Another, safer way to protect keywords (my preference) is to invoke m4 with the -P or --prefix-builtins option. Then, all builtin macro names are modified so they all start with the prefix m4_ and ordinary words are left alone. For example, using this option, one should write m4_define instead of define (as shown in the examples in this article).

The only trouble is that not all versions of m4 support this option - notably some PC versions under M$-DOS. Maybe that's just another reason to steer clear of hack code on M$-DOS and stay with Linux!

Contents

4.3 Gotcha 3 - Comments

Comments in m4 are introduced with the # character - everything from the # to the end of the line is ignored by m4 and simply passed unchanged to the output. If you want to use # in the HTML page then you would need to quote it like this - `#'. Another option (my preference) is to change the m4 comment character to something exotic like this: m4_changecom(`[[[[') and not have to worry about `#' symbols in your text.

If you want to use comments in the m4 file which do not appear in the final HTML file, then the macro m4_dnl (dnl = Delete to New Line) is for you. This suppresses everything until the next newline. 

m4_define(_NEWMACRO, `foo bar') m4_dnl This is a comment
Yet another way to have source code ignored is the m4_divert command. The main purpose of m4_divert is to save text in a temporary buffer for inclusion in the file later on - for example, in building a table of contents or index. However, if you divert to "-1" it just goes to limbo-land. This is useful for getting rid of the whitespace generated by the m4_define command, e.g.: 
m4_divert(-1) diversion on
m4_define(this ...)
m4_define(that ...)
m4_divert	diversion turned off

Contents

4.4 Gotcha 4 - Debugging

Another tip for when things go wrong is to increase the amount of error diagnostics that m4 emits. The easiest way to do this is to add the following to your m4 file as debugging commands: 
m4_debugmode(e)
m4_traceon
.
.
buggy lines
.
.
m4_traceoff

Contents

4.5 Gotcha 5 - perl in m4

There is a degree of syntax collision between perl, HTML and m4 and you need to be very careful in certain areas, particularly in the use of quotes. Perl's double quote mechanism (e.g. "this is $value\n") allows dynamic expansion of perl variables. It may help to use the more obscure equivalent form of ", namely the qq{ .... } construct which is identical (e.q. qq{this is $value\n}).

Of course perl's back quoting mechanism is also vulnerable and needs protection from m4. You can use qx{cmd} instead of the more usual `cmd` to run any system command cmd.

Contents

5. Conclusion

"ah ha!", I hear you say. "HTML 3.0 already has an include statement". Yes it has, and it looks like this: 
<!--#include file="junk.html" -->
The problem is that: Consequently, I don't use server-side includes.

There are several other features of m4 that I have not yet exploited in my HTML ramblings so far, such as regular expressions and doubtless many others. It might be interesting to create a "standard" stdlib.m4 for general use with nice macros for general text processing and HTML functions. By all means download my version of stdlib.m4 as a base for your own hacking. I would be interested in hearing of useful macros and if there is enough interest, maybe a Mini-HOWTO could evolve from this paper.

Beyond m4, there are many additional advantages in using Linux to develop HTML pages, far beyond the trivial help given by the typical WYSIWYG tools.

I hope you enjoy these little tricks and encourage you to contribute your own. Certainly, this little hacker will go on using m4 until HTML catches up.

Happy hacking!

6. Files to download

You can get the HTML and the m4 source code for this article here (for the sake of completeness, they're copylefted under GPL 2): 
using_m4.html	:this file
using_m4.m4.txt	:m4 source
stdlib.m4.txt	:Include file
makefile.txt

Contents

7. Bob Hepple

Bob Hepple has been hacking at Unix since 1981 under a variety of excuses and has somehow been paid for it at least some of the time. It's allowed him to pursue another interest - living in warm, exotic countries including Hong Kong, Australia, Qatar, Saudi Arabia, Lesotho and Singapore. His initial aversion to the cold was learned in the UK.

Contents

This site built with Linux
This page last updated on Mon Mar 25 10:28:28 EST 2002
For corrections/additions/suggestions for this page, please send email to: Bob Hepple

Copyright © 1997, 1998, 1999, 2000, 2001 Bob Hepple. All rights reserved.