CGI Wrapper

CGI Wrapper 0.2

Table of Contents

Synopsis
Description
      Overview
      Globals
      Errors / Uncaught Exceptions
      Configuration
      Running and Testing
      Administration
      Script Log
      Class-based CGIs
      Other File Types
      Cookies
      Subclassing CGI Wrapper
      Private Scripts
      Files
Release Notes
Limitations/Future
      To Do
Credit

The CGI Wrapper is a CGI script used to execute other Python CGI scripts. The wrapper provides convenient access to form fields and headers, exception catching, and usage and performance logging. Hooks are provided for cookies and class-based CGI scripts.

Description

The CGI Wrapper is a single CGI script used to execute other Python CGI scripts. A typical URL for a site that uses the wrapper looks like this:

http://www.somesite.com/server.cgi/SignUp

The server.cgi part is the CGI Wrapper and the SignUp part is the target Python script, whose real filename is SignUp.py. Also, that file is located in a directory named Scripts/, but there's no need for the user of the site to know about Scripts/ or .py; those are implementation details.

The wrapper provides the following benefits:

• Sets up global variables headers, fields, env and wrapper for use by the target script.
• Catches exceptions that the target script doesn't in order to provide a meaningful message to the web page reader and useful debugging information to the developer.
• Logs the date, time, duration and name of the target script for usage statistics and performance monitoring.
• Simplifies URLs by leaving out the extension and possibly the location of the script. This also hides the nature of the implementation of the page from the browser of your site.
• Provides a hook for integrating an object for cookies.
• Provides a hook for class-based CGIs.

You don't have to immediately write code to play with CGI Wrapper. There are several samples included. See Running and Testing below.

Globals

The globals set up by the wrapper for the target CGI script are:

Global	Type/Class	Description
headers	dictionary	Contains all the HTTP headers that will be sent back to the client. The default contents are 'Content-type': 'text/html'. Often, the headers don't need to be modified at all. One popular use of the headers is 'Redirect': 'someURL' to point the client to a different place.
fields	cgi.FieldStorage	This instance of FieldStorage comes from the standard Python cgi module. Typical uses include fields.has_key('someField') and fields['someField'].value. See the Python standard module documentation for cgi for more information.
environ	dictionary	This dictionary represents the environment variables passed to the CGI scripts. Scripts should use this rather than os.environ since future versions of CGI Wrapper could be tightly integrated into webservers, thereby changing the nature of how environment variables get passed around (e.g., no longer through the OS). Also, note that the environment may seem a little non-standard to the target CGI script since the web server is setting it up to run the CGI Wrapper instead. In most CGI scripts (that execute under the wrapper), the environment is not even needed.
wrapper	CGIWrapper	This is a pointer back to the CGI Wrapper instance. This allows CGI scripts to communicate with the wrapper if they want. However, this is hardly ever needed.
cookies	Cookie	This global is not set up by the wrapper, but is looked for upon exit of the CGI script. See the Cookies section below for more information.

Errors / Uncaught Exceptions

One of the main benefits of the wrapper is the handling of uncaught exceptions raised by target CGI scripts. The typical behavior of the wrapper upon detecting an uncaught exception is:

1. Log the time, error, script name and traceback to stderr. This information will typically appear in the web server's error log.
2. Display a web page containing an apologetic message to the user and useful debugging information for developers.
3. Save the above web page to a file so that developers can look at it after-the-fact. These HTML-based error messages are stored one-per-file, if the SaveErrorMessages setting is true (the default). They are stored in the directory named by the ErrorMessagesDir (defaults to 'ErrorMsgs').
4. Add an entry to the CGI Wrapper's error log, called Errors.csv.
5. E-mail the error message if the EmailErrors setting is true, using the settings ErrorEmailServer and ErrorEmailHeaders.

Here is a sample error page.

Archived error messages can be browsed through the administration page.

Error handling behavior can be configured as described in Configuration.

There are several configuration parameters through which you can alter how CGI Wrapper behaves. They are described below, including their default values:

ScriptsHomeDir   = '/Scripts'

This is where the wrapper always looks for the CGI scripts. There this location would not appear in URLs. The path can be relative to the CGI Wrapper's location, or an absolute path.

ChangeDir   = 1

If true, the current working directory is changed to the same directory as the target script. Otherwise, the current working directory is left alone and likely to be the same as the CGI Wrapper.

ExtraPaths   = []

A list of a strings which are inserted into sys.path. This setting is useful if you have one or more modules that are shared by your CGI scripts that expect to be able to import them.

ExtraPathsIndex   = 1

This is the index into sys.path where the ExtraPath value is inserted. Often the first path in sys.path is '.' which is why the default value of ExtraPathsIndex is 1.

LogScripts   = 1

If true, then the execution of each script is logged with useful information such as time, duration and whether or not an error occurred.

ScriptLogFilename   = 'Scripts.csv'

This is the name of the file that script executions are logged to. This value makes no difference if LogScripts is false/0. If the filename is not an absolute path, then it is relative to the directory of the CGI Wrapper.

ScriptLogColumns

= ['environ.REMOTE_ADDR', 'environ.REQUEST_METHOD', 'environ.REQUEST_URI', 'responseSize', 'scriptName', 'serverStartTimeStamp', 'serverDuration', 'scriptDuration', 'errorOccurred']

---

Specifies the columns that will be stored in the script log. Each column is the name of an attribute of CGI Wrapper. The Introspect CGI example gives a list of all CGI Wrapper attributes. Note that attributes which are dictionaries, UserDicts or subclasses of MiddleKit's KeyValueAccess class can have their attributes used through dot notation (e.g., obj1.obj2.attr).

ClassNames   = ['', 'Page']

This is the list of class names that CGI Wrapper looks for after executing a script. An empty string signifies a class whose name is the same as it's script (e.g., _admin in admin.py). See Class-based CGIs below.

ShowDebugInfoOnErrors   = 1

If true, then the uncaught exceptions will not only display a message for the user, but debugging information for the developer as well. This includes the traceback, HTTP headers, CGI form fields, environment and process ids.

UserErrorMessage

= 'The site is having technical difficulties with this page. An error has been logged, and the problem will be fixed as soon as possible. Sorry!'

---

This is the error message that is displayed to the user when an uncaught exception escapes the target CGI script.

ErrorLogFilename   = 'Errors.csv'

This is the name of the file where CGI Wrapper logs exceptions. Each entry contains the date & time, filename, pathname, exception name & data, and the HTML error message filename (assuming there is one).

SaveErrorMessages   = 1

If true, then errors (e.g., uncaught exceptions) will produce an HTML file with both the user message and debugging information. Developers/administrators can view these files after the fact, to see the details of what went wrong.

ErrorMessagesDir   = 'ErrorMsgs'

This is the name of the directory where HTML error messages get stored.

EmailErrors   = 0

If true, error messages are e-mail out according to the ErrorEmailServer and ErrorEmailHeaders settings. This setting defaults to false because the other settings need to be configured first.

ErrorEmailServer   = 'mail.-.com'

The SMTP server to use for sending e-mail error messages.

ErrorEmailHeaders

=

{
    'From':         '-@-.com',
    'To':           ['-@-.com'],
    'Reply-to':     '-@-.com',
    'Content-type': 'text/html',
    'Subject':      'Error'
}

---

The e-mail MIME headers used for e-mailing error messages. Be sure to configure 'From', 'To' and 'Reply-to' before using this feature.

You can override any of these values by creating a Config.dict file in the same directory as the wrapper and selectively specifying values in a dictionary like so:

{
    'ExtraPaths':       ['Backend', 'ThirdParty'],
    'ScriptLog':        'Logs/Scripts.csv'
}

Let's assume you have a webserver running on a UNIX box and a public HTML directory in your home directory. First, make a link from your public HTML directory to the source directory of the CGI Wrapper:

cd ~/public_html
ln -s ~/Projects/Webware/CGIWrapper pycgi

Note that in the Source directory there is a link to the Examples directory and that the CGI Wrapper will automatically look there (you can configure this; see Configuration). Therefore you can type a URL like:

http://localhost/~echuck/pycgi/server.cgi/Hello

Note that you didn't need to include Examples in the page or a .py at the end.

There is a special CGI example called Directory.py that lists the other examples as links you can click on to run or view source. In this way, you can see the full set of scripts.

http://localhost/~echuck/pycgi/server.cgi/Directory

The resulting page will look something like the following. (Note: Those links aren't real!)

Size	Script	View
96	Hello	view
167	Time	view
210	Error	view
565	View	view
802	Introspect	view
925	Colors	view
1251	Directory	view

The above instructions rely on your web server executing any files that end in .cgi. However, some servers require that executable scripts are also located in a special directory (such as cgi-bin) so you may need to take that into consideration when getting CGI Wrapper to work. Please consult your web server admin or your web server docs.

Administration

CGI Wrapper comes with a script for administration purposes. You can access it by specifying the _admin script in the URL. You typically only have to remember _admin because it contains links to the other scripts.

From the administration page, you can view the script log, the error log and the configuration of the server. The error log display also contains links to the archived error messages so that you can browse through them.

The adminstration scripts are good examples of class-based CGIs so you may wish to read through their code.

Here's an example of the admin page.

Script Log

The script log uses the comma-separated-value (CSV) format, which can be easily read by scripts, databases and spreadsheets. The file is located in the same directory as the CGI Wrapper. The columns are fairly self-explanatory especially once you look at actual file. The Configuration section has more details under the ScriptLogColumns setting.

Class-based CGIs

As you write CGI scripts, and especially if they are for the same site, you may find that they have several things in common. For example, the generated pages may all have a common toolbar, heading and/or footing. You might also find that you display programmatically collected data in a similiar fashion throughout your pages.

When you see these kinds of similarities, it's time to start designing a class hierarchy that takes advantage of inheritance, encapsulation and polymorphism in order to save you from duplicative work.

For example your base class could have methods header(), body() and footer() with the header and footer being fully implemented. Subclasses would then only need to override body() and would therefore inherit their look and feel from one source. You could take this much farther by providing several utility methods in the base class that are available to subclasses for use or customization.

CGI Wrapper provides a hook to support such class-based CGI scripts by checking for a classes in the target script. The ClassNames setting, whose default value is ['', 'Page'], controls this behavior. After a script executes, CGI Wrapper checks these classes. The empty string is a special case which specifies a class whose name is the same name as it's containing script (e.g., the class _admin in the script _admin.py).

If a matching class is found, it is automatically instantiated so that you don't have to do so in every script. The instantiation is basically:

print TheClass(info).html()

Where info has keys ['wrapper', 'fields', 'environ', 'headers'].

A good example of class-based CGIs are the admin pages for CGI Wrapper. Start by reading AdminPage.py and then continuing with the various admin scripts such as _admin.py and _showConfig.py. All of these are located in the same directory as CGI Wrapper.

On a final note, if you find that you're developing a sophisticated web-based application with accounts, sessions, persistence, etc. then you should consider using the WebKit, which is analagous to Apple's WebObjects and Sun's Java Servlets.

Other File Types

CGI Wrapper assumes that a URL with no extension (such .html) is a Python script. However, if the URL does contain an extension, the wrapper simply passes it through via HTTP redirection (e.g., the Location: header).

This becomes important when one of your CGI scripts writes a relative URL to a non-CGI resource. Such a relative URL ends up forcing server.cgi to come into play.

Cookies

Cookies are often an important part of web programming. CGI Wrapper does not provide explicit support for cookies, however, it provides an easily utilized hook for them.

If upon completion, the target script has a cookies global variable, the CGI Wrapper will print it to stdout. This fits in nicely with the Cookie module written by Timothy O'Malley. You will find a copy of this module in the WebUtils package of Webware for Python. You could also find it by searching for "Cookie" at the Vaults of Parnassus located at http://www.vex.net/parnassus/.

Subclassing CGI Wrapper

This is just a note that CGI Wrapper is a class with well defined, broken-out methods. If it doesn't behave precisely as you need, you may very well be able to subclass it and override the appropriate methods. See the source which contains numerous doc strings and comments.

Private Scripts

Any script starting with an underscore ('_') is considered private to CGI Wrapper and is expected to be found in the CGI Wrapper's directory (as opposed to the directory named by the ScriptsHomeDir setting).

The most famous private script is the admin script which then contains links to others:

http://localhost/~echuck/pycgi/server.cgi/_admin

A second script is _dumpCSV which dumps the contents of a CSV file (such as the script log or the error log).

Files

server.cgi - Just a Python script with a generic name (that appears in URLs) that imports CGIWrapper.py. By keeping the CGIWrapper.py file separate we get byte code caching (CGIWrapper.pyc) and syntax highlighting when viewing or editing the script.

CGIWrapper.py - The main script that does the work.

Config.dict - An optional file containing a dictionary that overrides default configuration settings.

Scripts.csv - The log of script executions as described above.

Errors.csv - The log of uncaught exceptions including date & time, script filename and archived error message filename.

ErrorMsgs/Error-scriptname-YYYY-MM-DD-*.py - Archived error messages.

_*.py - Administration scripts for CGI Wrapper.

Release Notes

• 0.2 - 5/12/2000
  • handleException()
    • Added settings: ErrorLogFilename, SaveErrorMessages, ErrorMessagesDir.
    • Added error logging to 'Errors.csv'.
    • HTML error message pages are now saved to a file that can reviewed later by the developer/administrator.
    • Broke out the functionality of handleException() into methods that can be overriden.
  • Renamed ErrorMessage setting to UserErrorMessage.
  • Added "e-mail errors" feature.
  • Logs now get a header row (with column titles).
  • The Scripts.csv log can now be configured by the ScriptLogColumns setting which is an array of names.
  • Removed special __info__ URL and added _admin page which shows basic info and a menu for the script log, error log and configuration. Added _dumpCSV and _config. Created AdminPage base class for all admin scripts.
  • The integration with class-based CGIs can now be configured with the 'ClassNames' setting which is an array of names attempted in order. The empty string means the same name as the script.
  • Found a bad glitch: a relative URL that points to a non-script file that is embedded in a page from a CGI script will end up hitting "server.cgi" which then barfs on the non-script. I fixed up CGI Wrapper. The new policy is that anything with an extension is re-served with a "Location:" header. Anything without an extension is assumed to be a Python script.
  • sys.exit(0), which throws an exception, is now detected and handled gracefully. i.e., the wrapper no longer reports it as an error when a target CGI script does this.
  • Various code clean up and documentation improvements.

Note: CGI scripts are fine for small features, but if you're developing a full blown web-based application then you typically want more support, persistence and classes. That's where other Webware components like WebKit and MiddleKit come into play.

Here are some future ideas, with no committments or timelines as to when/if they'll be realized. This is open source, so feel free to jump in!

The following are in approximate order of the author's perceived priority, but the numbering is mostly for reference.

To Do

1. Examples: Make a Cookie example. (In the meantime, just see the main doc string of Cookie.py in WebUtils.)
2. Wrapper: When a script produces no output, the CGI Wrapper should report that problem. (This most often happens for class based CGIs with incorrect class names.)
3. There should probably be an option to clear the output of a script that raised an uncaught exception. Sometimes that could help in debugging.
4. Admin: Create a summary page for the script and error logs.
5. Wrapper: It's intended that the CGIWrapper class could be embedded in a server and a single instance reused several times. The class is not quite there yet.
6. Wrapper: CGI scripts never get cached as byte code (.pyc) which would provide a performance boost. 2000-04-17 ce: Check out py_compile module.
7. Wrapper: The error log columns should be configurable just like the script log columns.
8. Code review: Misc functions towards bottom of CGIWrapper
9. Code review: Use of _realStdout and sys.stdout on multiple serve() calls.
10. Wrapper: Create a subclass of Python's CGI server that uses CGIWrapper. This would include caching the byte code in memory.
11. Wrapper: htmlErrorPageFilename() uses a "mostly works" technique that could be better. See source.
12. Wrapper: Keep a list of file extensions (such as .py .html .pl) mapped to their handlers. When processing a URL, iterate through the list until a file with that extension is found, then serve it up through it's handler.
13. There's no password protection on the administration scripts.
14. Provide integration (and therefore increased performance) with web servers such as Apache and AOLserver.
15. Wrapper: If the script log can't be opened for writing, nothing happens. What's the best action here?
16. Wrapper: Error e-mails are always in HTML format. It may be useful to have a plain text version for those with more primitive e-mail clients.

Author: Chuck Esterbrook

The idea of a CGI wrapper is based on a WebTechniques article by Andrew Kuchling.