A crucial part of the OneFS SmartQuotas system is to provide user notifications regarding quota enforcement violations, both when a violation event occurs and while violation state persists on a scheduled basis.
An enforcement quota may have several notification rules associated with it. Each notification rule specifies a condition and an action to be performed when the condition is met. Notification rules are considered part of enforcements. Clearing an enforcement also clears any notification rules associated with it.
Enforcement quotas support the following notification settings:
|Quota Notification Setting
|Uses the global default notification for the specified type of quota.
|Custom – basic
|Enables creation of basic custom notifications that apply to a specific quota. Can be configured for any or all the threshold types (hard, soft, or advisory) for the specified quota.
|Custom – advanced
|Enables creation of advanced, custom notifications that apply to a specific quota. Can be configured for any or all of the threshold types (hard, soft, or advisory) for the specified quota.
|Disables all notifications for the quota.
A quota notification condition is an event which may trigger an action defined by a notification rule. These notification rules may specify a schedule (for example, “every day at 5:00 AM”) for performing an action or immediate notification of a certain condition. Examples of notification conditions include:
- Notify when a threshold is exceeded; at most, once every 5 minutes
- Notify when allocation is denied; at most, once an hour
- Notify while over threshold, daily at 2 AM
- Notify while grace period expired weekly, on Sundays at 2 AM
Notifications are triggered for events grouped by the following two categories:
|Includes the write-denied notification triggered when a hard threshold denies a write and the threshold-exceeded notification, triggered at the moment a hard, soft, or advisory threshold is exceeded. These are one-time notifications because they represent a discrete event in time.
|Generated on a scheduled basis to indicate a persisting condition, such as a hard, soft, or advisory threshold being over a limit or a soft threshold’s grace period being expired for a prolonged period.
Each notification rule can perform either one or none of the following notification actions.
|Quota Notification Action
|Sends an alert for one of the quota actions, detailed below.
|Email Manual Address
|Sends email to a specific address, or multiple addresses (OneFS 8.2 and later).
|Emails an owner mapping based on its identity source.
The email owner mapping is as follows:
|Lookup is performed against the domain controller (DC). If the user does not have an email setting, a configurable transformation from user name and DC fully qualified domain name is performed in order to generate an email address.
|LDAP user email resolution is similar to AD users. In this case, only the email attribute looked up in the LDAP server is configurable by an administrator based on the LDAP schema for the user account information.
|Only the configured email transformation for the NIS fully qualified domain name is used.
|Only the configured email transformation is used.
The actual quota notification is handled by a daemon, isi_quota_notify_d, which performs the following functions:
- Processes kernel notification events that get sent out. They are matched to notification rules to generate instant notifications (or other actions as specified in the notification rule)
- Processes notification schedules – The daemon will check notification rules on a scheduled basis. These rules specify what violation condition should trigger a notification on a regular scheduled basis.
- Performs notifications based on rule configuration to generate email messages or alert notifications.
- Manages persistent notification states so that pending events are processed in the event of a restart.
- Handles rescan requests when quotas are created or modified
SmartQuotas provides email templates for advisory, grace, and regular notification configuration, which can be found under /etc/ifs. The advisory limit email template (/etc/ifs/quota_email_advisory_template.txt) for example, displays:
Subject: Disk quota exceeded The <ISI_QUOTA_DOMAIN_TYPE> quota on path <ISI_QUOTA_PATH> owned by <ISI_QUOTA_OWNER> has exceeded the <ISI_QUOTA_TYPE> limit. The quota limit is <ISI_QUOTA_THRESHOLD>, and <ISI_QUOTA_USAGE> is currently in use. <ISI_QUOTA_HARD_LIMIT> Contact your system administrator for details.
An email template contains text, and, optionally, variables that represent quota values. The following table lists the SmartQuotas variables that may be included in an email template.
|Quota type. Valid values are: directory, user, group, default-directory, default-user, default-group
|Expiration date of grace period
|Fri Jan 8 12:34:56 PST 2021
|Grace period, in days
|Includes the hard limit information of the quota to make advisory/soft email notifications more informational.
|You have 30 MB left until you reach the hard quota limit of 50 MB.
|Hostname of the node on which the quota event occurred
|Name of quota domain owner
|Path of quota domain
|Disk space in use
Note that the default quota templates under /etc/ifs send are configured to send email notifications with a plain text MIME type. However, editing a template to start with an HTML tag (<html>) will allow an email client to interpret and display it as HTML content. For example:
<html><Body> <h1>Quota Exceeded</h1><p></p> <hr> <p> The path <ISI_QUOTA_PATH> has exceeded the threshold <ISI_QUOTA_THRESHOLD> for this <ISI_QUOTA_TYPE> quota. </p> </body></html>
Various system alerts are sent out to the standard cluster Alerting system when specific events occur. These include:
|An attempt to process a notification rule failed externally, such as an undelivered email.
|A notification rule failed due to a configuration issue, such as a non-existent user or missing email address.
|A child quota’s advisory/soft/hard limit is greater than any of parent quota’s hard limit.
|A quota threshold was exceeded. The conditions under which this alert is triggered are defined by notification rules.
|An invariant was violated that resulted in a forced domain rescan.