Customizing the service engine

This is an advanced topic intended for those who wish to try their hand at extending the internal engine in WeeWX. Before attempting these examples, you should be reasonably proficient with Python.

Warning

Please note that the API to the service engine may change in future versions!

At a high level, WeeWX consists of an engine that is responsible for managing a set of services. A service consists of a Python class which binds its member functions to various events. The engine arranges to have the bound member function called when a specific event happens, such as a new LOOP packet arriving.

The services are specified in lists in the [Engine][[Services]] stanza of the configuration file. The [[Services]] section lists all the services to be run, broken up into different service lists.

These lists are designed to orchestrate the data as it flows through the WeeWX engine. For example, you want to make sure that data has been processed by the quality control service, StdQC, before putting them in the database. Similarly, the reporting system must come after the data has been put in the database. These groups ensure that things happen in the proper sequence.

See the table The standard WeeWX services for a list of the services that are normally run.

Modifying an existing service

The service weewx.engine.StdPrint prints out new LOOP and archive packets to the console when they arrive. By default, it prints out the entire record, which generally includes a lot of possibly distracting information and can be rather messy. Suppose you do not like this, and want it to print out only the time, barometer reading, and the outside temperature whenever a new LOOP packet arrives.

This could be done by subclassing the default print service StdPrint and overriding member function new_loop_packet().

Create the file user/myprint.py:

from weewx.engine import StdPrint
from weeutil.weeutil import timestamp_to_string

class MyPrint(StdPrint):

    # Override the default new_loop_packet member function:
    def new_loop_packet(self, event):
        packet = event.packet
        print("LOOP: ", timestamp_to_string(packet['dateTime']),
            "BAR=",  packet.get('barometer', 'N/A'),
            "TEMP=", packet.get('outTemp', 'N/A'))

This service substitutes a new implementation for the member function new_loop_packet. This implementation prints out the time, then the barometer reading (or N/A if it is not available) and the outside temperature (or N/A).

You then need to specify that your print service class should be loaded instead of the default StdPrint service. This is done by substituting your service name for StdPrint in service_list, located in [Engine]/[[Services]]:

[Engine]
    [[Services]]
        ...
        report_services = user.myprint.MyPrint, weewx.engine.StdReport

Note that the report_services must be all on one line. Unfortunately, the parser ConfigObj does not allow options to be continued on to following lines.

Creating a new service

Suppose there is no service that can be easily customized for your needs. In this case, a new one can easily be created by subclassing off the abstract base class StdService, and then adding the functionality you need. Here is an example that implements an alarm, which sends off an email when an arbitrary expression evaluates True.

This example is included in the standard distribution as examples/alarm.py:

import logging
import smtplib
import socket
import threading
import time
from email.mime.text import MIMEText

import weewx
from weeutil.weeutil import timestamp_to_string, option_as_list
from weewx.engine import StdService

log = logging.getLogger(__name__)


# Inherit from the base class StdService:
class MyAlarm(StdService):
    """Service that sends email if an arbitrary expression evaluates true"""

    def __init__(self, engine, config_dict):
        # Pass the initialization information on to my superclass:
        super().__init__(engine, config_dict)

        # This will hold the time when the last alarm message went out:
        self.last_msg_ts = 0

        try:
            # Dig the needed options out of the configuration dictionary.
            # If a critical option is missing, an exception will be raised and
            # the alarm will not be set.
            self.expression    = config_dict['Alarm']['expression']
            self.time_wait     = int(config_dict['Alarm'].get('time_wait', 3600))
            self.timeout       = int(config_dict['Alarm'].get('timeout', 10))
            self.smtp_host     = config_dict['Alarm']['smtp_host']
            self.smtp_user     = config_dict['Alarm'].get('smtp_user')
            self.smtp_password = config_dict['Alarm'].get('smtp_password')
            self.SUBJECT       = config_dict['Alarm'].get('subject', 
                                                          "Alarm message from weewx")
            self.FROM          = config_dict['Alarm'].get('from',
                                                          'alarm@example.com')
            self.TO            = option_as_list(config_dict['Alarm']['mailto'])
        except KeyError as e:
            log.info("No alarm set.  Missing parameter: %s", e)
        else:
            # If we got this far, it's ok to start intercepting events:
            self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)    # 1
            log.info("Alarm set for expression: '%s'", self.expression)

    def new_archive_record(self, event):
        """Gets called on a new archive record event."""

        # To avoid a flood of nearly identical emails, this will do
        # the check only if we have never sent an email, or if we haven't
        # sent one in the last self.time_wait seconds:
        if (not self.last_msg_ts 
                or abs(time.time() - self.last_msg_ts) >= self.time_wait):
            # Get the new archive record:
            record = event.record

            # Be prepared to catch an exception in the case that the expression 
            # contains a variable that is not in the record:
            try:                                                            # 2
                # Evaluate the expression in the context of the event archive 
                # record. Sound the alarm if it evaluates true:
                if eval(self.expression, None, record):                     # 3
                    # Sound the alarm! Launch in a separate thread,
                    # so it doesn't block the main LOOP thread:
                    t = threading.Thread(target=MyAlarm.sound_the_alarm, 
                                         args=(self, record))
                    t.start()
                    # Record when the message went out:
                    self.last_msg_ts = time.time()
            except NameError as e:
                # The record was missing a named variable. Log it.
                log.info("%s", e)

    def sound_the_alarm(self, record):
        """Sound the alarm in a 'try' block"""

        # Wrap the attempt in a 'try' block so we can log a failure.
        try:
            self.do_alarm(record)
        except socket.gaierror:
            # A gaierror exception is usually caused by an unknown host
            log.critical("Unknown host %s", self.smtp_host)
            # Reraise the exception. This will cause the thread to exit.
            raise
        except Exception as e:
            log.critical("Unable to sound alarm. Reason: %s", e)
            # Reraise the exception. This will cause the thread to exit.
            raise

    def do_alarm(self, record):
        """Send an email out"""

        # Get the time and convert to a string:
        t_str = timestamp_to_string(record['dateTime'])

        # Log the alarm
        log.info('Alarm expression "%s" evaluated True at %s' 
                 % (self.expression, t_str))

        # Form the message text:
        msg_text = 'Alarm expression "%s" evaluated True at %s\nRecord:\n%s' \
                   % (self.expression, t_str, str(record))
        # Convert to MIME:
        msg = MIMEText(msg_text)

        # Fill in MIME headers:
        msg['Subject'] = self.SUBJECT
        msg['From']    = self.FROM
        msg['To']      = ','.join(self.TO)

        try:
            # First try end-to-end encryption
            s = smtplib.SMTP_SSL(self.smtp_host, timeout=self.timeout)
            log.debug("Using SMTP_SSL")
        except (AttributeError, socket.timeout, socket.error) as e:
            log.debug("Unable to use SMTP_SSL connection. Reason: %s", e)
            # If that doesn't work, try creating an insecure host, 
            # then upgrading
            s = smtplib.SMTP(self.smtp_host, timeout=self.timeout)
            try:
                # Be prepared to catch an exception if the server
                # does not support encrypted transport.
                s.ehlo()
                s.starttls()
                s.ehlo()
                log.debug("Using SMTP encrypted transport")
            except smtplib.SMTPException as e:
                log.debug("Using SMTP unencrypted transport. Reason: %s", e)

        try:
            # If a username has been given, assume that login is required 
            # for this host:
            if self.smtp_user:
                s.login(self.smtp_user, self.smtp_password)
                log.debug("Logged in with user name %s", self.smtp_user)

            # Send the email:
            s.sendmail(msg['From'], self.TO,  msg.as_string())
            # Log out of the server:
            s.quit()
        except Exception as e:
            log.error("SMTP mailer refused message with error %s", e)
            raise

        # Log sending the email:
        log.info("Email sent to: %s", self.TO)


if __name__ == '__main__':
    """This section is used to test alarm.py. It uses a record and alarm
     expression that are guaranteed to trigger an alert.

     You will need a valid weewx.conf configuration file with an [Alarm]
     section that has been set up as illustrated at the top of this file."""

    from optparse import OptionParser
    import weecfg
    import weeutil.logger

    usage = """Usage: python alarm.py --help    
       python alarm.py [CONFIG_FILE|--config=CONFIG_FILE]

Arguments:

      CONFIG_PATH: Path to weewx.conf """

    epilog = """You must be sure the WeeWX modules are in your PYTHONPATH. 
    For example:

    PYTHONPATH=/home/weewx/bin python alarm.py --help"""

    # Force debug:
    weewx.debug = 1

    # Create a command line parser:
    parser = OptionParser(usage=usage, epilog=epilog)
    parser.add_option("--config", dest="config_path", metavar="CONFIG_FILE",
                      help="Use configuration file CONFIG_FILE.")
    # Parse the arguments and options
    (options, args) = parser.parse_args()

    try:
        config_path, config_dict = weecfg.read_config(options.config_path, args)
    except IOError as e:
        exit("Unable to open configuration file: %s" % e)

    print("Using configuration file %s" % config_path)

    # Set logging configuration:
    weeutil.logger.setup('alarm', config_dict)

    if 'Alarm' not in config_dict:
        exit("No [Alarm] section in the configuration file %s" % config_path)

    # This is a fake record that we'll use
    rec = {'extraTemp1': 1.0,
           'outTemp': 38.2,
           'dateTime': int(time.time())}

    # Use an expression that will evaluate to True by our fake record.
    config_dict['Alarm']['expression'] = "outTemp<40.0"

    # We need the main WeeWX engine in order to bind to the event, 
    # but we don't need for it to completely start up. So get rid of all
    # services:
    config_dict['Engine']['Services'] = {}
    # Now we can instantiate our slim engine, using the DummyEngine class...
    engine = weewx.engine.DummyEngine(config_dict)
    # ... and set the alarm using it.
    alarm = MyAlarm(engine, config_dict)

    # Create a NEW_ARCHIVE_RECORD event
    event = weewx.Event(weewx.NEW_ARCHIVE_RECORD, record=rec)

    # Use it to trigger the alarm:
    alarm.new_archive_record(event)

This service expects all the information it needs to be in the configuration file weewx.conf in a new section called [Alarm]. So, add the following lines to your configuration file:

[Alarm]
    expression = "outTemp < 40.0"
    time_wait = 3600
    smtp_host = smtp.example.com
    smtp_user = myusername
    smtp_password = mypassword
    mailto = auser@example.com, anotheruser@example.com
    from   = me@example.com
    subject = "Alarm message from WeeWX!"

There are three important highlighted points to be noted in this example.

1. (Line 45) Here is where the binding happens between an event, weewx.NEW_ARCHIVE_RECORD, and a member function, self.new_archive_record. When the event NEW_ARCHIVE_RECORD occurs, the function self.new_archive_record will be called. There are many other events that can be intercepted. Look in the file weewx/_init_.py.

2. (Line 61) Some hardware do not emit all possible observation types in every record, so it's possible that a record may be missing some types that are used in the expression. This try block will catch the NameError exception that would be raised should this occur.

3. (Line 64) This is where the test is done for whether to sound the alarm. The [Alarm] configuration options specify that the alarm be sounded when outTemp < 40.0 evaluates True, that is when the outside temperature is below 40.0 degrees. Any valid Python expression can be used, although the only variables available are those in the current archive record.

Another example expression could be:

expression = "outTemp < 32.0 and windSpeed > 10.0"

In this case, the alarm is sounded if the outside temperature drops below freezing and the wind speed is greater than 10.0.

Note that units must be the same as whatever is being used in your database, that is, the same as what you specified in option target_unit.

Option time_wait is used to avoid a flood of nearly identical emails. The new service will wait this long before sending another email out.

Email will be sent through the SMTP host specified by option smtp_host. The recipient(s) are specified by the comma separated option mailto.

Many SMTP hosts require user login. If this is the case, the user and password are specified with options smtp_user and smtp_password, respectively.

The last two options, from and subject are optional. If not supplied, WeeWX will supply something sensible. Note, however, that some mailers require a valid "from" email address and the one that WeeWX supplies may not satisfy its requirements.

To make this all work, you must first copy the alarm.py file to the user directory. Then tell the engine to load this new service by adding the service name to the list report_services, located in [Engine]/[[Services]]:

[Engine]
   [[Services]]
        report_services = weewx.engine.StdPrint, weewx.engine.StdReport, user.alarm.MyAlarm

Again, note that the option report_services must be all on one line — the ConfigObj parser does not allow options to be continued on to following lines.

In addition to this example, the distribution also includes a low-battery alarm (lowBattery.py), which is similar, except that it intercepts LOOP events instead of archiving events.

Adding a second data source

A very common problem is wanting to augment the data from your weather station with data from some other device. Generally, you have two approaches for how to handle this:

• Run two instances of WeeWX, each using its own database and weewx.conf configuration file. The results are then combined in a final report, using WeeWX's ability to use more than one database. See the Wiki entry How to run multiple instances of WeeWX for details on how to do this.

• Run one instance, but use a custom WeeWX service to augment the records coming from your weather station with data from the other device.

This section covers the latter approach.

Suppose you have installed an electric meter at your house, and you wish to correlate electrical usage with the weather. The meter has some sort of connection to your computer, allowing you to download the total power consumed. At the end of every archive interval you want to calculate the amount of power consumed during the interval, then add the results to the record coming off your weather station. How would you do this?

Here is the outline of a service that retrieves the electrical consumption data and adds it to the archive record. It assumes that you already have a function download_total_power() that, somehow, downloads the amount of power consumed since time zero.

File user/electricity.py

import weewx
from weewx.engine import StdService

class AddElectricity(StdService):

    def __init__(self, engine, config_dict):

      # Initialize my superclass first:
      super(AddElectricity, self).__init__(engine, config_dict)

      # Bind to any new archive record events:
      self.bind(weewx.NEW_ARCHIVE_RECORD, self.new_archive_record)

      self.last_total = None

    def new_archive_record(self, event):

        total_power = download_total_power()

        if self.last_total:
            net_consumed = total_power - self.last_total
            event.record['electricity'] = net_consumed

        self.last_total = total_power

This adds a new key electricity to the record dictionary and sets it equal to the difference between the amount of power currently consumed and the amount consumed at the last archive record. Hence, it will be the amount of power consumed over the archive interval. The unit should be Watt-hours.

As an aside, it is important that the function download_total_power() does not delay very long because it will sit right in the main loop of the WeeWX engine. If it's going to cause a delay of more than a couple seconds you might want to put it in a separate thread and feed the results to AddElectricity through a queue.

To make sure your service gets run, you need to add it to one of the service lists in weewx.conf, section [Engine], subsection [[Services]].

In our case, the obvious place for our new service is in data_services. When you're done, your section [Engine] will look something like this:

 #   This section configures the internal WeeWX engine.

[Engine]

    [[Services]]
        # This section specifies the services that should be run. They are
        # grouped by type, and the order of services within each group
        # determines the order in which the services will be run.
        xtype_services = weewx.wxxtypes.StdWXXTypes, weewx.wxxtypes.StdPressureCooker, weewx.wxxtypes.StdRainRater, weewx.wxxtypes.StdDelta
        prep_services = weewx.engine.StdTimeSynch
        data_services = user.electricity.AddElectricity
        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate
        archive_services = weewx.engine.StdArchive
        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS
        report_services = weewx.engine.StdPrint, weewx.engine.StdReport