Swift Middleware

My work on Audit Watchers has brought me into learning about and creating new Swift Middleware! Here are some of the things I've discovered during that process.

Middleware

Swift supports use of Python WSGI (Web Server Gateway Interface) middleware to process requests and responses that go through Swift nodes including the proxy-server, account-servers, container-servers and object-servers. This kind of middleware exists in swift as both optional and natively active modules that affect how Swift WSGI servers behave.

Middleware is added and configured using Python Paste configuration files. Most often this will be through the Proxy Server since this is the node through which all requests are received, but I have been working mostly with the middleware for account-servers and object-servers.

Here is an example configuration file for a SAIO Account Server (/etc/swift/account-server/1.conf):

[DEFAULT]
devices = /srv/1/node
mount_check = false
disable_fallocate = true
bind_ip = 127.0.0.1
bind_port = 6012
workers = 1
user = <your-user-name>
log_facility = LOG_LOCAL2
recon_cache_path = /var/cache/swift
eventlet_debug = true

[pipeline:main]
pipeline = recon account-server

[app:account-server]
use = egg:swift#account

[filter:recon]
use = egg:swift#recon

[account-replicator]
rsync_module = {replication_ip}::account{replication_port}

[account-auditor]

[account-reaper]

Active middleware is listed under [pipeline:main] and each section starting [filter: ... ] and [app: ...] is a configuration for middleware.

The use = egg:swift#foobar lines are PasteDeploy entrypoints. Entrypoints are references to Python objects in packages that are named, require certain arguments, and expect a specific return value.

This is achieved using Paste factories, paste.filter_factory in particular. Filter refers to filter factories (which, given an application, return a “filtered” version of that application) and [app: ...] refers to app factories (which return a WSGI application). These factories are the means to modify the server behaviour.

Examples of Swift Middleware include:

  • CatchErrors - Middleware that provides high-level error handling and ensures that a transaction id will be set for every request.
  • Healthcheck - Healthcheck middleware is used for monitoring. It returns a 200 response with “OK” in the body.
  • Recon - Middleware used for monitoring and telemetry. A request using /recon/load|mem|async... will return various system metrics.

Creating and Extending Middleware

I've been creating new middleware, Audit Watchers for Accounts, and extending existing middleware, Swift Recon. The new middleware, swift/common/middleware/audit_watcher.py is a means to allow operators (maintainers of swift systems) to hook into auditor processes and run custom code on each account in a cluster. Potentially though, this middleware could be used to do the same for objects and containers. The addition to Swift Recon is a use-case for the previously mentioned auditor hooks, a simple telemetry watcher to record the accounts that exist and their location in the cluster.

The entrypoints installed by default are stored in the swift/setup.cfg file, and installed using alongside the other Swift requirements. To add new natively supported middleware required modifications to the default configuration files.

In /swift/setup.cfg

....

[entry_points]
paste.app_factory =
    proxy = swift.proxy.server:app_factory
    object = swift.obj.server:app_factory
    mem_object = swift.obj.mem_server:app_factory
    container = swift.container.server:app_factory
    account = swift.account.server:app_factory

paste.filter_factory =
    healthcheck = swift.common.middleware.healthcheck:filter_factory
    crossdomain = swift.common.middleware.crossdomain:filter_factory
    ....
    xprofile = swift.common.middleware.xprofile:filter_factory
    versioned_writes = swift.common.middleware.versioned_writes:filter_factory

swift.account_audit_watcher =
    account_watcher = swift.common.middleware.audit_watcher:filter_factory

....

In /etc/swift/account-server/1.conf ... 4.conf

[account-auditor]
watchers = account_watcher

When an account auditor is initialised, any watchers specified in these configuration files are created and attached to auditor.watchers. A watcher is a class with at least these four methods:

  • __init__(self, conf, logger)
  • start(self, audit_type)
  • see_target(self, *data)
  • end(self)

The auditor will call watcher.start(audit_type) at the start of an audit pass, watcher.see_target(*data) for each account audited, and watcher.end() at the end of an audit pass. The user-supplied watcher code runs entirely in a child process of the auditor process (of which there can be many, if parallel audits are in use). This lets the auditor continue working even if the watcher crashes or hangs. These sub-processes are automatically killed (after a short delay) once an audit pass is completed.

The most up-to-date implementation of the code itself can be viewed here:
Let developers/operators add watchers to account audit - https://review.openstack.org/#/c/268830


Sources

  1. http://docs.openstack.org/developer/swift/middleware.html
  2. http://docs.openstack.org/developer/swift/development_middleware.html
  3. http://docs.openstack.org/developer/swift/development_saio.html
  4. https://review.openstack.org/#/c/268830