Auditors In Swift

What is an Auditor?

In Swift Auditors are daemons (background processes) that ensure data integrity. There are auditors for accounts, containers, and objects. An object auditor will detect data corruption in files and move any affected file to quarantine allowing replication processes to replace it with a good copy. Auditors also log any errors they encounter during crawling, in the case of account and container auditors, making sure all account and container databases are valid.

How I can use Auditors in my Project

Because Auditors are an existing set of Swift processes that have similar file exploration and analysis behaviour to that of Slogging, they seem ideal to extend or to use as the basis for a new stats gatherer. One big advantage is that using an existing auditor process to gather data adds only small overhead compared to the IO cost of running an additional filesystem-walker process.

As before, my primary focus is on Account stats. An Account Auditor looks specifically for database files (.db extension) in Swift's mounted devices and checks whether they are account databases and whether the number of containers, objects, and bytes used checks out against the statistics according to policy totals.

An existing patch by Samuel Merritt makes use of hooks to allow operators (users of Swift) to add Watchers that can carry out tasks on audited objects. I was able to adapt his code on object auditors to work with account auditors, along with a new Account Watcher class to log encountered account databases to file.

To use these patches requires installation of the Basic-Audit-Watcher package via setuptools using:

python setup.py install

The target Object Auditor configuration file/s also need to be updated with a comma-separated list of watcher entry points as previously defined in the package install. The Watcher included in the Basic-Audit-Watcher package is called "basic".

[object-auditor]
watchers = basic

The basic Object Watcher outputs .auditlog files to the /tmp directory.

How it Works

Watchers are classes defined by operators and can do potentially anything. Watchers are wrapped up in a subprocess to isolate the user-supplied code from the auditor code which helps protect the auditor against user-derived code issues that could slow the process down and cause problems, like memory leaks.

Watcher code is carried out in three main functions:

  • start(audit_type) - called at the beginning of an audit.
  • see_item() - called when the corresponding audit item is discovered, eg. an object or container.
  • end() - called at the completion of an audit.

Sources

  1. Swift Architectural Overview
  2. Swift Patch - Let operators add watchers to object audit
  3. Account Auditor Code (swift/account/auditor.py)