AWK As A Major Systems Programming Language

I started this paper in 2013, and in 2015 sent it out for review to the people listed later on. After incorporating comments, I sent it to Rik Farrow, the editor of the USENIX magazine ;login: to see if he would publish it. He declined to do so, for reasonably good reasons.

The paper languished, forgotten, until early 2018 when I came across it and decided to polish it off, put it up on GitHub, and make it available from my home page in HTML.

If you are interested in language design and evolution in general, and in Awk in particular, I hope you will enjoy reading this paper. If not, then why are you bothering looking at it now?

1 Introduction

At the March 1991 USENIX conference, Henry Spencer presented a paper entitled AWK As A Major Systems Programming Language. In it, he described his experiences using the original version of awk to write two significant “systems” programs—a clone for a reasonable subset of the nroff formatter¹, and a simple parser generator.

He described what awk did well, as well as what it didn’t, and presented a list of things that awk would need to acquire in order to take the position of a reasonable alternative to C for systems programming tasks on Unix systems.

In particular, awk lies about in the middle of the spectrum between C, which is “close to the metal,” and the shell, which is quite high-level. A language at this level that is useful for doing systems programming is very desirable.

This paper reviews Henry’s wish list, and describes some of the events that have occurred in the Unix/Linux world since 1991. It presents a case that gawk—GNU Awk—fills most of the major needs Henry listed way back in 1991, and then describes the author’s opinion as to why other languages have successfully filled the systems programming role which awk did not. It discusses how the current version of gawk may finally be able to join the ranks of other popular, powerful, scripting languages in common use today, and ends off with some counter-arguments and the author’s responses to them.

Acknowledgements

Thanks to Andrew Schorr, Henry Spencer, Nelson H.F. Beebe, and Brian Kernighan for reviewing an earlier draft of this paper.

2 That Was Then …

In this section we review the state of the Unix world in 1991, as well as the state of awk, and then list what Henry Spencer saw as missing for awk.

2.1 The Unix World in 1991

Undoubtedly, many readers of this paper were not using computers in 1991, so this section provides the context in which Henry’s paper was written. In March of 1991:

2.2 What Awk Lacked In 1991

Here is a summary of what was wrong with the awk picture in 1991. These are in the same order as presented Henry’s paper. We qualify each issue in order to later discuss how it has been addressed over time.

In private email, Henry added the following items, saying “there are a couple more things I’d add now, in hindsight.” These are direct quotes:

The first of these is somewhere between a language design and a language implementation issue. The latter is a language design issue.

3 … And This Is Now

3.1 What Awk Has Today

3.2 And What GNU Awk Has Today

The more difficult of the quality of implementation issues are addressed by gawk. In particular:

With the 4.1 release, all three versions (gawk, pgawk, and dgawk) are merged into a single executable, considerably reducing the required installation “footprint.”

While gawk has almost always been faster than Brian Kernighan’s awk, recent performance improvements bring it closer to mawk’s performance level (a byte-code based execution engine and internal improvements in array indexing).

And gawk clearly has the most features of any version, many of which considerably increase the power of the language.

3.3 So Where Does Awk Stand?

Despite all of the above, gawk is not as popular as other scripting languages. Since 1991, we can point to four major scripting languages which have enjoyed, or currently enjoy, differing levels of popularity: PERL, tcl/tk, Python, and Ruby. We think it is fair to say that Python and Ruby are the most popular scripting languages in the second decade of the 21st century.

Is awk, as we’ve described it up to this point, now ready to compete with the other languages? Not quite yet.

4 Key Reasons Why Other Languages Have Gained Popularity

In retrospect, it seems clear (at least to us!) that there are two major reasons that all of the previously mentioned languages have enjoyed significant popularity. The first is their extensibility. The second is namespace management.

One certainly cannot attribute their popularity to improved syntax. In the opinion of many, PERL and Ruby both suffer from terrible syntax. Tcl’s syntax is readable but nothing special. Python’s syntax is elegant, although slightly unusual. The point here is that they all differ greatly in syntax, and none really offers the clean pattern–action paradigm that is awk’s trademark, yet they are all popular languages.

If not syntax, then what? We believe that their popularity stems from the fact that all of these languages are easily extensible. This is true with both “modules” in the scripting language, and more importantly, with access to C level facilities via dynamic library loading.

Furthermore, these languages allow you to group related functions and variables into packages or modules: they let you manage the namespace.

awk, on the other hand, has always been closed. An awk program cannot even change its working directory, much less open a connection to an SQL database or a socket to a server on the Internet somewhere (although gawk can do the latter).

If one examines the number of extensions available for PERL on CPAN, or for Python such as PyQt or the Python tk bindings, it becomes clear that extensibility is the real key to power (and from there to popularity).

Furthermore, in awk, all global variables and functions share a single namespace. This prevents many good software development practices based on the principle of information hiding.

To summarize: A reasonable language definition, efficient implementations, debuggers and profilers are necessary but not sufficient for true power. The final ingredients are extensibility and namespaces.

5 Filling The Extensibility Gap

With version 4.1, gawk (finally) provides a defined C API for extending the core language.

5.1 API Overview

The API makes it possible to write functions in C or C++ that are callable from an awk program as if the function were written in awk. The most straightforward way to think of these functions is as user-defined functions that happen to be implemented in a different language.

5.2 Discussion

Considerable thought went into the design of the API. The gawk documentation provides a full description of the API itself, with examples (over 50 pages worth!), as well as some discussion of the goals and design decisions behind the API (in an appendix). The development was done over the course of about a year and a half, together with the developers of xgawk, a fork of gawk that added features that made using extensions easier, and included an extension for processing XML files in a way that fit naturally with the pattern–action paradigm. While it may not be perfect, the gawk developers feel that it is a good start.

FIXME: Henry Spencer suggests adding more info on the API and on the design decisions. I think this paper is long enough, and the full doc is quite big. It’d be hard to pull API doc into this paper in a reasonable fashion, although it would be possible to review some of the design decisions. Comments?

The major xgawk additions to the C code base have been merged into gawk, and the extensions from that project have been rewritten to use the new API. As a result, the xgawk project developers renamed their project gawkextlib, and the project now provides only extensions.⁵

It is notable that functions written in awk can do a number of things that extension functions cannot, such as modify any variables, do I/O, call awk built-in functions, and call other user-defined functions.

While it would certainly be possible to provide APIs for all of these features for extension functions, this seemed to be overkill. Instead, the gawk developers took the view that extension functions should provide access to external facilities, and provide communication to the awk level via function parameters and/or global variables, including associative arrays, which are the only real data structure.

Consider a simple example. The standard du program can recursively walk one or more arbitrary file hierarchies, call stat() to retrieve file information, and then sum up the blocks used. In the process, du must track hard links, so that no file is accounted for or reported more than once.

The ‘filefuncs’ extension shipped with gawk provides a stat() function that takes a pathname and fills in an associative array with the information retrieved from stat(). The array elements have names like "size", "mtime" and so on, with corresponding appropriate values. (Compare this to PERL’s stat() function that returns a linearly-indexed array!)

The fts() function in the ‘filefuncs’ extension builds on stat() to create a multidimensional array of arrays that describes the requested file hierarchies, with each element being an array filled in by stat(). Directories are arrays containing elements for each directory entry, with an element named "." for the array itself.

Given that fts() does the heavy lifting, du can be written quite nicely, and quite portably⁶, in awk. See du in awk, for the code, which weighs in at under 250 lines. Much of this is comments and argument parsing.

5.3 Future Work

The extension facility is relatively new, and undoubtedly has introduced new “dark corners” into gawk. These remain to be uncovered and any new bugs need to be shaken out and removed.

Some issues are known and may not be resolvable. For example, 64-bit integer values such as the timestamps in stat() data on modern systems don’t fit into awk’s 64-bit double-precision numbers which only have 53 bits of significand. This is also a problem for the bit-manipulation functions.

With respect to namespaces, in 2017 I (finally) figured out how namespaces in awk ought to work to provide the needed functionality while retaining backwards compatibility. The code is currently in the feature/namespaces branch of gawk’s Git repository. It will eventually be merged into the master branch for release as part of gawk 5.0.

6 Counterpoints

Brian Kernighan raised several counterpoints in response to an earlier draft of the paper. They are worth addressing (or at least trying to):

I think that Perl’s original inclusion of most of the Unix system calls was, from a language design standpoint, ultimately a mistake. At the time it was first done, there was no other choice: dynamic loading of libraries didn’t exist on Unix systems in the early and mid-1980s (nor did shared libraries, for that matter). But having all those built-in functions bloats the language, making it harder to learn, document, and maintain, and I definitely did not wish to go down that path for gawk.

With respect to Python, the question is: how are those libraries implemented? Are they built-in to the interpreter and separated from the “core” language simply by the language design? Or are they dynamically loaded modules?

If the latter, that sounds like an argument for the case of having extensions, not against it. And indeed, this merely emphasizes the point made at the end of the previous section, which is that to make an extension facility really scalable, you also need some sort of namespace / module capability.

Thus, Brian is correct: an extension facility is needed, but the last part of the puzzle would be a module facility in the language. I think that I have solved this, and invite the curious reader to checkout the branch named earlier and provide feedback.

This is hard to argue with. Nonetheless, gawk’s --lint option may be of help here, as well as the --dump-variables option which produces a list of all variables used in the program.

Since my Python experience is minimal, I have little to say here; it might be that if I were more familiar with Python, I would start using it for small scripts instead of awk.

On the other hand, with discipline, it’s possible to write fairly good-sized, understandable and maintainable awk programs; in my experience awk does scale up well beyond the one-liner range.

Not to mention that Brian published a whole book of awk programs larger than one line. :-) (See the Resources section.)

My definition of “small” and “big” has changed over time. 250 lines may be big for a script, but the du.awk program is much smaller than a full implementation in C: GNU du is over 1,100 lines of C, plus all the libraries it relies upon in the GNU Coreutils.

With respect to shorter examples, nothing springs to mind immediately. However, gawk comes with several useful extensions that are worth exploring, much more than we’ve covered here.

For example, the readdir extension in the gawk distribution causes gawk to read directories and return one record per directory entry in an easy-to-parse format:

Also, the gawkextlib project provides some very interesting extensions. Of particular interest are the XML and JSON extensions, but there are a number of others, and it’s worth checking out.

In short, it’s too early to really tell. This is the beginning of an experiment. I hope it will be a fun journey for me, the other gawk maintainers, and the larger community of awk users.

7 Conclusion

It has taken much longer than any awk fan would like, but finally, GNU Awk fills in almost all the gaps listed by Henry Spencer for awk to be really useful as a systems programming language.

In addition, experience from other popular languages has shown that extensibility and namespaces are the keys to true power, usability, and popularity.

With the release of gawk 4.1, we feel that gawk (and thus the Awk language) are now almost on par with the basic capabilities of other popular languages. With gawk 5.0, we hope to truly reach par.

Is it too late in the game? If enough people start to write extensions for gawk, then perhaps awk will return to the scripting language limelight. If not, then the gawk developers will have wasted a lot of time and effort. (We hope not!) Time will tell.

For now though, we hope that this paper will have piqued your curiosity, and that you will take the time to give gawk a fresh look.

Appendix A Resources

Appendix B Awk Code For du

Here is the du program, written in Awk. Besides demonstrating the power of the stat() and fts() extensions and gawk’s multidimensional arrays, it also shows the switch statement and the built-in bit manipulation functions and(), or(), and compl().

The output is not identical to GNU du’s, since filenames are not sorted. However, gawk’s built-in sorting facilities should make sorting the output straightforward; we leave that as the traditional “exercise for the reader.”

#! /usr/local/bin/gawk -f

# du.awk --- write POSIX du utility in awk.
# See http://pubs.opengroup.org/onlinepubs/9699919799/utilities/du.html
#
# Most of the heavy lifting is done by the fts() function in the "filefuncs"
# extension.
#
# We think this conforms to POSIX, except for the default block size, which
# is set to 1024. Following GNU standards, set POSIXLY_CORRECT in the
# environment to force 512-byte blocks.
#
# Arnold Robbins
# arnold@skeeve.com

@include "getopt"
@load "filefuncs"

BEGIN {
    FALSE = 0
    TRUE = 1

    BLOCK_SIZE = 1024   # Sane default for the past 30 years
    if ("POSIXLY_CORRECT" in ENVIRON)
        BLOCK_SIZE = 512        # POSIX default

    compute_scale()

    fts_flags = FTS_PHYSICAL
    sum_only = FALSE
    all_files = FALSE

    while ((c = getopt(ARGC, ARGV, "aHkLsx")) != -1) {
        switch (c) {
        case "a":
            # report size of all files
            all_files = TRUE;
            break
        case "H":
            # follow symbolic links named on the command line
            fts_flags = or(fts_flags, FTS_COMFOLLOW)
            break
        case "k":
            BLOCK_SIZE = 1024       # 1K block size
            break
        case "L":
            # follow all symbolic links

            # fts_flags &= ~FTS_PHYSICAL
            fts_flags = and(fts_flags, compl(FTS_PHYSICAL))

            # fts_flags |= FTS_LOGICAL
            fts_flags = or(fts_flags, FTS_LOGICAL)
            break
        case "s":
            # do sums only
            sum_only = TRUE
            break
        case "x":
            # don't cross filesystems
            fts_flags = or(fts_flags, FTS_XDEV)
            break
        case "?":
        default:
            usage()
            break
        }
    }

    # if both -a and -s
    if (all_files && sum_only)
        usage()

    for (i = 0; i < Optind; i++)
        delete ARGV[i]

    if (Optind >= ARGC) {
        delete ARGV     # clear all, just to be safe
        ARGV[1] = "."   # default to current directory
    }

    fts(ARGV, fts_flags, filedata)  # all the magic happens here

    # now walk the trees
    if (sum_only)
        sum_walk(filedata)
    else if (all_files)
        all_walk(filedata)
    else
        top_walk(filedata)
}

# usage --- print a message and die

function usage()
{
    print "usage: du [-a|-s] [-kx] [-H|-L] [file] ..." > "/dev/stderr"
    exit 1
}

# compute_scale --- compute the scale factor for block size calculations

function compute_scale(     stat_info, blocksize)
{
    stat(".", stat_info)

    if (! ("devbsize" in stat_info)) {
        printf("du.awk: you must be using filefuncs extension from gawk 4.1.1 or later\n") > "/dev/stderr"
        exit 1
    }

    # Use "devbsize", which is the units for the count of blocks
    # in "blocks".
    blocksize = stat_info["devbsize"]
    if (blocksize > BLOCK_SIZE)
        SCALE = blocksize / BLOCK_SIZE
    else    # I can't really imagine this would be true
        SCALE = BLOCK_SIZE / blocksize
}

# islinked --- return true if a file has been seen already

function islinked(stat_info,        device, inode, ret)
{
    device = stat_info["dev"]
    inode = stat_info["ino"]

    ret = ((device, inode) in Files_seen)

    return ret
}

# file_blocks --- return number of blocks if a file has not been seen yet

function file_blocks(stat_info,     device, inode)
{
    if (islinked(stat_info))
        return 0

    device = stat_info["dev"]
    inode = stat_info["ino"]

    Files_seen[device, inode]++

    return block_count(stat_info)   # delegate actual counting
}

# block_count --- return number of blocks from a stat() result array

function block_count(stat_info,     result)
{
    if ("blocks" in stat_info)
        result = int(stat_info["blocks"] / SCALE)
    else
        # otherwise round up from size
        result = int((stat_info["size"] + (BLOCK_SIZE - 1)) / BLOCK_SIZE)

    return result
}

# sum_dir --- data on a single directory

function sum_dir(directory, do_print,   i, sum, count)
{
    for (i in directory) {
        if ("." in directory[i]) {  # directory
            count = sum_dir(directory[i], do_print)
            count += file_blocks(directory[i]["."])
            if (do_print)
                printf("%d\t%s\n", count, directory[i]["."]["path"])
        } else {            # regular file
            count = file_blocks(directory[i]["stat"])
        }
        sum += count
    }

    return sum
}

# simple_walk --- summarize directories --- print info per parameter

function simple_walk(filedata, do_print,    i, sum, path)
{
    for (i in filedata) {
        if ("." in filedata[i]) {   # directory
            sum = sum_dir(filedata[i], do_print)
            path = filedata[i]["."]["path"]
        } else {            # regular file
            sum = file_blocks(filedata[i]["stat"])
            path = filedata[i]["path"]
        }
        printf("%d\t%s\n", sum, path)
    }
}

# sum_walk --- summarize directories --- print info only for the top set of directories

function sum_walk(filedata)
{
    simple_walk(filedata, FALSE)
}

# top_walk --- data on the main arguments only

function top_walk(filedata)
{
    simple_walk(filedata, TRUE)
}

# all_walk --- data on every file

function all_walk(filedata, i, sum, count)
{
    for (i in filedata) {
        if ("." in filedata[i]) {   # directory
            count = all_walk(filedata[i])
            sum += count
            printf("%s\t%s\n", count, filedata[i]["."]["path"])
        } else {            # regular file
            if (! islinked(filedata[i]["stat"])) {
                count = file_blocks(filedata[i]["stat"])
                sum += count
                if (i != ".")
                    printf("%d\t%s\n", count, filedata[i]["path"])
            }
        }
    }
    return sum
}

AWK As A Major Systems Programming Language — Revisited

Preface

1 Introduction

Acknowledgements

2 That Was Then …

2.1 The Unix World in 1991

2.2 What Awk Lacked In 1991

3 … And This Is Now

3.1 What Awk Has Today

3.2 And What GNU Awk Has Today

3.3 So Where Does Awk Stand?

4 Key Reasons Why Other Languages Have Gained Popularity

5 Filling The Extensibility Gap

5.1 API Overview

5.2 Discussion

5.3 Future Work

6 Counterpoints

7 Conclusion

Appendix A Resources

Appendix B Awk Code For `du`

Footnotes

(1)

(2)

(3)

(4)

(5)

(6)

AWK As A Major Systems Programming Language — Revisited

Preface

1 Introduction

Acknowledgements

2 That Was Then …

2.1 The Unix World in 1991

2.2 What Awk Lacked In 1991

3 … And This Is Now

3.1 What Awk Has Today

3.2 And What GNU Awk Has Today

3.3 So Where Does Awk Stand?

4 Key Reasons Why Other Languages Have Gained Popularity

5 Filling The Extensibility Gap

5.1 API Overview

5.2 Discussion

5.3 Future Work

6 Counterpoints

7 Conclusion

Appendix A Resources

Appendix B Awk Code For du

Footnotes

(1)

(2)

(3)

(4)

(5)

(6)

Appendix B Awk Code For `du`