This is a reference guide to the executable utilities that come with WeeWX:

wee_config for changing the configuration file, and configuring new device drivers;
wee_database for reconfiguring the database;
wee_debug for producing debug reports for remote support;
wee_device for configuring your hardware;
wee_extension for installing and removing extensions;
wee_import for importing historical data from external sources;
wee_reports for running reports without running WeeWX itself;
weewxd the main weewx program;
wunderfixer for resending data missing on the Weather Underground site.

wee_config

When you install WeeWX for the first time, the installation process will prompt you for the most essential options, such as the type of hardware you are using, latitude, longitude, or altitude. But, what if you want to change these later? In particular, what if you want to change device drivers? You could edit the definitive configuration file, weewx.conf, by hand — described in detail in the User's Guide — but it's a big file with lots of nuances. Alternatively, if you're just changing something simple, you may be able to use the utility wee_config.

Before starting, it's worth running it with the --help option to see how it is used:

wee_config --help

This results in:

Usage: wee_config --help
       wee_config --version
       wee_config --list-drivers
       wee_config --reconfigure CONFIG_FILE|--config=CONFIG_FILE
           [--driver=DRIVER]  [--units=(us|metric)]
           [--latitude=yy.y] [--longitude=xx.x] [--altitude=zz.z,(foot|meter)]
           [--location="Home Sweet Home"] [--register=(y,n)]
           [--output=OUT_CONFIG] [--no-prompt] [--no-backup] [--verbosity=N]
       wee_config --install --dist-config=DIST_CONFIG --output=OUT_CONFIG
           [--driver=DRIVER]  [--units=(us|metric)]
           [--latitude=yy.y] [--longitude=xx.x] [--altitude=zz.z,(foot|meter)]
           [--location="Home Sweet Home"] [--register=(y,n)]
           [--no-prompt] [--no-backup] [--verbosity=N]
       wee_config --upgrade CONFIG_FILE|--config=CONFIG_FILE
           --dist-config=DIST_CONFIG
           [--output=OUT_CONFIG] [--no-prompt] [--no-backup]
           [--warn-on-error] [--verbosity=N]

User actions:

--version       Show the WeeWX version, then exit.
--list-drivers  List the available weewx device drivers, then exit.
--reconfigure   Modify an existing configuration file CONFIG_FILE with any
                specified station parameters.  Use this command with the
                --driver option to change the device driver.

System actions (not normally done by users):

--install       Install a new configuration file starting with the contents of
                DIST_CONFIG, prompting for station parameters.
--upgrade       Update the contents of configuration file CONFIG_FILE to the
                installed version, then merge the result with the contents of
                configuration file DIST_CONFIG.

Station parameters:

  --driver      --units
  --latitude    --longitude
  --altitude    --location
  --register


Options:
  -h, --help            show this help message and exit
  --version             Show the weewx version then exit.
  --list-drivers        List the available device drivers.
  --reconfigure         Reconfigure an existing configuration file.
  --install             Install a new configuration file.
  --upgrade             Update an existing configuration file, then merge with
                        contents of DIST_CONFIG.
  --config=CONFIG_FILE  Use configuration file CONFIG_FILE.
  --dist-config=DIST_CONFIG
                        Use template configuration file DIST_CONFIG.
  --output=OUT_CONFIG   Save to configuration file OUT_CONFIG.  If not
                        specified then replace existing configuration file.
  --driver=DRIVER       Use the driver DRIVER. For example,
                        weewx.drivers.vantage
  --latitude=yy.y       The station latitude in decimal degrees.
  --longitude=xx.x      The station longitude in decimal degrees.
  --altitude=zz,(foot|meter)
                        The station altitude in either feet or meters. For
                        example, '750,foot' or '320,meter'
  --location=LOCATION   A text description of the station. For example,
                        "Santa's workshop, North Pole"
  --units=(metric|us)   Set display units to 'metric' or 'us'.
  --register=(y/n)      Register this station in the weewx registry?
  --no-prompt           Do not prompt. Use default or specified values.
  --no-backup           When replacing an existing configuration file, do not
                        create a backup copy.
  --warn-on-error       Only warn if an update is not possible.  Default
                        behavior is to warn then exit.
  --debug               Show diagnostic information while running.
  --verbosity=N         How much status to display, 0-3

Actions and options

Option --config

The utility is pretty good about "guessing" where the configuration file weewx.conf is, but if you've done an unusual install, you may have to tell it explicitly. You can do this by either putting the location directly in the command line:

wee_config /home/weewx/weewx.conf

or by using option --config:

wee_config --config=/home/weewx/weewx.conf

Action --list-drivers

Use this action to list which device drivers are available on your system. For example:

wee_config --list-drivers

This will result in something like:

Module name              Driver name    Version  Status
  weewx.drivers.acurite    AcuRite        0.19     No module named usb
  weewx.drivers.cc3000     CC3000         0.3
  weewx.drivers.fousb      FineOffsetUSB  1.7      No module named usb
  weewx.drivers.simulator  Simulator      3.0
  weewx.drivers.te923      TE923          0.14     No module named usb
  weewx.drivers.ultimeter  Ultimeter      0.13
  weewx.drivers.vantage    Vantage        3.0
  weewx.drivers.wmr100     WMR100         3.0      No module named usb
  weewx.drivers.wmr9x8     WMR9x8         3.0
  weewx.drivers.ws1        WS1            0.19
  weewx.drivers.ws23xx     WS23xx         0.24
  weewx.drivers.ws28xx     WS28xx         0.34     No module named usb

The column Status can give you some indication of whether you are missing any modules to use this driver. It's not completely accurate, but works for most drivers.

Action --reconfigure

This action is used to change station parameters, including the device driver. The reconfigure action will prompt for all of the required station parameters.

wee_config --reconfigure

When used with the --no-prompt option, reconfigure will modify specific parameters with no interaction. For example, this would set the station altitude:

wee_config --reconfigure --altitude=35,foot --no-prompt

This would change the driver to a user-installed netatmo driver:

wee_config --reconfigure --driver=user.netatmo --no-prompt

Example: changing the driver

Say that you originally installed WeeWX with the Simulator. Now you've bought a Davis Vantage and you want to switch to that. Here's how you do it. First, use the action --list-drivers to see which drivers are installed.

wee_config --list-drivers

Module name              Driver name    Version  Status
  weewx.drivers.acurite    AcuRite        0.19     
  weewx.drivers.cc3000     CC3000         0.3
  weewx.drivers.fousb      FineOffsetUSB  1.7      
  weewx.drivers.simulator  Simulator      3.0
  weewx.drivers.te923      TE923          0.14     
  weewx.drivers.ultimeter  Ultimeter      0.13
  weewx.drivers.vantage    Vantage        3.0
  weewx.drivers.wmr100     WMR100         3.0      
  weewx.drivers.wmr9x8     WMR9x8         3.0
  weewx.drivers.ws1        WS1            0.19
  weewx.drivers.ws23xx     WS23xx         0.24
  weewx.drivers.ws28xx     WS28xx         0.34

From the list, you will find that the name of the driver for the Vantage is weewx.drivers.vantage. Now run wee_config, with the --reconfigure action, specifying that driver:

wee_config --reconfigure --driver=weewx.drivers.vantage

You can also run without specifying the driver a priori:

wee_config --reconfigure

The utility will go through your previously selected options, such as station description, latitude, longitude, altitude, etc.. With the exception of your newly specified device driver, all your previously selected values will be the defaults. So, all you have to do is keep hitting enter. This is what it looked like when I switched from the simulator to the Vantage driver:

Using configuration file /home/weewx/weewx.conf
Enter a brief description of the station, such as its location.  For example:
Santa's Workshop, North Pole
description [My Little Town, Oregon]:

Specify altitude, with units 'foot' or 'meter'.  For example:
35, foot
12, meter
altitude [700, foot]:

Specify latitude in decimal degrees, negative for south.
latitude [0.00]:
Specify longitude in decimal degrees, negative for west.
longitude [0.00]:

You can register your station on weewx.com, where it will be included
in a map. You will need a unique URL to identify your station (such as a
website, or WeatherUnderground link).
Include station in the station registry (y/n)? [y]:
Unique URL: [http://weewx.com]:

Indicate the preferred units for display: ['us', 'metric', 'metricwx']
units [us]:

Installed drivers include:
  0) AcuRite         (weewx.drivers.acurite)
  1) CC3000          (weewx.drivers.cc3000)
  2) FineOffsetUSB   (weewx.drivers.fousb)
  3) Simulator       (weewx.drivers.simulator)
  4) TE923           (weewx.drivers.te923)
  5) Ultimeter       (weewx.drivers.ultimeter)
  6) Vantage         (weewx.drivers.vantage)
  7) WMR100          (weewx.drivers.wmr100)
  8) WMR300          (weewx.drivers.wmr300)
  9) WMR9x8          (weewx.drivers.wmr9x8)
 10) WS1             (weewx.drivers.ws1)
 11) WS23xx          (weewx.drivers.ws23xx)
 12) WS28xx          (weewx.drivers.ws28xx)
choose a driver [3]:
Saved backup to /home/weewx/weewx.conf.20200328054724

If this is too much trouble, you can specify the --no-prompt option:

wee_config --reconfigure --driver=weewx.drivers.vantage --no-prompt

This will accept all the defaults, including your new device driver, without asking for any input.

Action --upgrade

Use this action to upgrade a configuration file, such as weewx.conf. Normally, this is done automatically by the installer, but there are occasions when it must be done manually. In particular, this can happen when you run several instances of weewxd on the same machine and each has its own, unique, configuration file. The one named weewx.conf will get upgraded, but not the others.

Say you have an additonal configuration file, foo.conf, located in /etc/weewx/foo.conf. Here's how you can upgrade it:

First, locate the "distribution configuration file." This is the version of weewx.conf that came with the new version of WeeWX.

If you used the setup.py install method, it will be in /home/weewx/weewx.conf.X.Y.Z, where X.Y.Z is the version number.

If you used a package installer, it will be in /etc/weewx/weewx.conf.dist.
Next use the utility to upgrade it:
```
sudo wee_config --upgrade --config=/etc/weewx/foo.conf --dist-config=path-to-dist-config
```
where path-to-dist-config is the path to the distribution configuration file identified above.

Your old version of foo.conf will be copied to a timestamped backup. Then, it will upgraded and the results will replace the old copy.

The option --upgrade is idempotent. That is, subsequent invocations do not change the result (although they will leave multiple backups behind).

wee_database

This database utility simplifies typical database maintenance operations. For example, it can rebuild the daily summaries or check a SQLite database for embedded strings where floats are expected.

Run the utility with the --help option to see how it is used:

wee_database --help

This will result in an output that looks something like this:

Usage: wee_database --help
       wee_database --create
       wee_database --reconfigure
       wee_database --transfer --dest-binding=BINDING_NAME [--dry-run]
       wee_database --add-column=NAME [--type=(REAL|INTEGER)]
       wee_database --rename-column=NAME --to-name=NEW_NAME
       wee_database --drop-columns=NAME1,NAME2,...
       wee_database --check
       wee_database --update [--dry-run]
       wee_database --drop-daily
       wee_database --rebuild-daily [--date=YYYY-mm-dd |
                                    [--from=YYYY-mm-dd] [--to=YYYY-mm-dd]]
                                    [--dry-run]
       wee_database --reweight [--date=YYYY-mm-dd |
                               [--from=YYYY-mm-dd] [--to=YYYY-mm-dd]]
                               [--dry-run]
       wee_database --calc-missing [--date=YYYY-mm-dd |
                                   [--from=YYYY-mm-dd[THH:MM]] [--to=YYYY-mm-dd[THH:MM]]]
       wee_database --check-strings
       wee_database --fix-strings [--dry-run]

Description:

Manipulate the WeeWX database. Most of these operations are handled
automatically by WeeWX, but they may be useful in special cases.

Options:
  -h, --help            show this help message and exit
  --create              Create the WeeWX database and initialize it with the
                        schema.
  --reconfigure         Create a new database using configuration information
                        found in the configuration file. In particular, the
                        new database will use the unit system found in option
                        [StdConvert][target_unit]. The new database will have
                        the same name as the old database, with a '_new' on
                        the end.
  --transfer            Transfer the WeeWX archive from source database to
                        destination database.
  --add-column=NAME     Add new column NAME to database.
  --type=TYPE           New database column type (INTEGER|REAL) (option --add-
                        column only).  Default is 'REAL'.
  --rename-column=NAME  Rename the column with name NAME.
  --to-name=NEW_NAME    New name of the column (option --rename-column only).
  --drop-columns=NAME1,NAME2,...
                        Drop one or more columns. Names must be separated by
                        commas, with NO SPACES.
  --check               Check the calculations in the daily summary tables.
  --update              Update the daily summary tables if required and
                        recalculate the daily summary maximum windSpeed
                        values.
  --calc-missing        Calculate and store any missing derived observations.
  --check-strings       Check the archive table for null strings that may have
                        been introduced by a SQL editing program.
  --fix-strings         Fix any null strings in a SQLite database.
  --drop-daily          Drop the daily summary tables from a database.
  --rebuild-daily       Rebuild the daily summaries from data in the archive
                        table.
  --reweight            Recalculate the weighted sums in the daily summaries.
  --config=CONFIG_FILE  Use configuration file CONFIG_FILE.
  --date=YYYY-mm-dd     This date only (options --calc-missing and --rebuild-
                        daily only).
  --from=YYYY-mm-dd[THH:MM]
                        Start with this date or date-time (options --calc-
                        missing and --rebuild-daily only).
  --to=YYYY-mm-dd[THH:MM]
                        End with this date or date-time (options --calc-
                        missing and --rebuild-daily only).
  --binding=BINDING_NAME
                        The data binding to use. Default is 'wx_binding'.
  --dest-binding=BINDING_NAME
                        The destination data binding (option --transfer only).
  --dry-run             Print what would happen but do not do it. Default is
                        False.

Actions and options

Action --create

If the database does not already exist, this action will create it and initialize it with the schema specified in the WeeWX configuration file. Because WeeWX does this automatically, this action is rarely needed.

wee_database --create

Action --reconfigure

This action is useful for changing the schema or unit system in your database.

It creates a new database with the same name as the old, except with the suffix _new attached at the end (nominally, weewx.sdb_new if you are using SQLite, weewx_new if you are using MySQL). It then initializes it with the schema specified in weewx.conf. Finally, it copies over the data from your old database into the new database.

wee_database --reconfigure

See the section Changing the database unit system in an existing database in the Customization Guide for step-by-step instructions that use this option.

Action --transfer

This action is useful for moving your database from one type of database to another, such as from SQLite to MySQL. To use it, you must have two bindings specified in your weewx.conf configuration file. One will serve as the source, the other as the destination. Specify the source binding with option --binding, the destination binding with option --dest-binding. The --binding option may be omitted in which case the default wx-binding will be used.

wee_database --transfer --binding=source_binding --dest-binding=dest_binding
wee_database --transfer --dest-binding=dest_binding

See the Wiki for examples of moving data from SQLite to MySQL, and from MySQL to SQLite, using wee_database.

Action --add-column

This action adds a new database observation type (column) to the database. If used without the --type option, the type will default to REAL.

wee_database --add-column

Optionally, option --type can be used with a SQL type REAL, INTEGER, or any other SQL column definition (such as INTEGER DEFAULT 0).

wee_database --add-column=NAME --type=TYPE

Action --rename-column

Use this action to rename a database observation type (column) to a new name. It requires the option --to-name.

wee_database --rename-column=NAME --to-name=NEW_NAME

For example, to rename the column luminosity in your schema to illuminance:

wee_database --rename-column=luminosity --to-name=illuminance

Action --drop-columns

This action will drop one or more observation types (columns) from the database. If more than one column name is given, they should be separated by commas and no spaces.

It is an error to attempt to drop a non-existing column. In this case, nothing will be done.

wee_database --drop-columns=NAME1,NAME2

NOTE: When dropping columns from a SQLite database, the entire database must be copied except for the dropped columns. Because this can be quite slow, if you are dropping more than one column, it is better to do them all in one pass. This is why option --drop-columns accepts more than one name.

Action --check

This action will check the calculations in the daily summary tables as well as checking the archive for null strings (refer to --check-strings). If the daily summary tables contain summaries calculated using an old algorithm, the user is advised to update the daily summary tables using the --update action. If null strings are found the user is advised to fix them using the --fix-strings action.

wee_database --check

Action --update

This action updates the daily summary tables to use interval weighted calculations as well as recalculating the windSpeed maximum daily values and times. Interval weighted calculations are only applied to the daily summaries if not previously applied. The update process is irreversible and users are advised to backup their database before performing this action.

wee_database --update

For further information on interval weighting and recalculation of daily windSpeed maximum values, see the sections Changes to daily summaries and Recalculation of windSpeed maximum values in the Upgrade Guide.

Action --drop-daily

In addition to the regular archive data, every WeeWX database also includes a daily summary table for each observation type. Because there can be dozens of observation types, there can be dozens of these daily summaries. It doesn't happen very often, but there can be occasions when it's necessary to drop them all and then rebuild them. Dropping them by hand would be very tedious! This action does them all at once.

wee_database --drop-daily

Action --rebuild-daily

This action is the inverse of action --drop-daily in that it rebuilds the daily summaries from the archive data. In most cases it is not necessary to drop the daily summary tables using the action --drop-daily before rebuilding them. The action --rebuild-daily accepts a number of date related options, --date, --from and --to that allow selective rebuilding of the daily summaries for one or more days rather than for the entire archive history. These options may be useful if bogus data has been removed from the archive covering a single day or a period of few days. The daily summaries can then be rebuilt for this period only, resulting in a faster rebuild and detailed low/high values and the associated times being retained for unaffected days. The --date option limits the daily summary rebuild to the specified date. The --from and --to options may be used together or individually to limit the daily summary rebuild to a specified period. When used individually the --to option limits the rebuild to the inclusive period from the earliest date for which there is data in the database through to and including the specified date. Similarly when used individually the --from option limits the rebuild to the inclusive period from the specified date through until the last date for which there is data in the database. When the --from and --to options are used together the daily summary rebuild is limited to the specified inclusive period.

wee_database --rebuild-daily
wee_database --rebuild-daily --date=YYYY-mm-dd
wee_database --rebuild-daily --from=YYYY-mm-dd
wee_database --rebuild-daily --to=YYYY-mm-dd
wee_database --rebuild-daily --from=YYYY-mm-dd --to=YYYY-mm-dd

Note
Whilst the --from and --to options will accept optional hour and minutes in the format THH:MM, any such hour and minute options are ignored by the --rebuild action as the daily summaries represent whole days only and it is not possible to partially rebuild a daily summary.

Note
When used with the --rebuild-daily action the period defined by --to and --from is inclusive and the daily summary tables will be rebuild for the day defined by --from and the day defined by --to and all days in between.

Action --reweight

As an alternative to dropping and rebuilding the daily summaries, this action simply rebuilds the weighted daily sums (used to calculate averages) from the archive data. It does not touch the highs and lows. It is much faster than --rebuild-daily, and has the advantage that the highs and lows remain unchanged.

Other options are as in --rebuild-daily.

Action --calc-missing

This action calculates derived observations for archive records in the database and then stores the calculated observations in the database. This can be useful if erroneous archive data is corrected or some additional observational data is added to the archive that may alter previously calculated or missing derived observations.

The period over which the derived observations are calculated can be limited through use of the --date, --from and/or --to options. When used without any of these options --calc-missing will calculate derived observations for all archive records in the database. The --date option limits the calculation of derived observations to the specified date only. The --from and --to options can be used together to specify the start and end date-time respectively of the period over which derived observations will be calculated. If --from is used by itself the period is fom the date-time specified up to and including the last archive record in the database. If --to is used by itself the period is the first archive record in the database through to the specified date-time.

wee_database --calc-missing
wee_database --calc-missing --date=YYYY-mm-dd
wee_database --calc-missing --from=YYYY-mm-dd[THH:MM]
wee_database --calc-missing --to=YYYY-mm-dd[THH:MM]
wee_database --calc-missing --from=YYYY-mm-dd[THH:MM] --to=YYYY-mm-dd[THH:MM]

Note
When a YYYY-mm-dd date is used as the --from option the period used by --calc-missing includes records after midnight at the start of YYYY-mm-dd, if a YYYY-mm-ddTHH:MM format date-time is used as the --from option the period includes records after YYYY-mm-dd HH:MM. When a YYYY-mm-dd date is used as the --to option the period includes records up to and including midnight at the end of YYYY-mm-dd, if a YYYY-mm-ddTHH:MM format date-time is used as the --to option the period includes records up to and including YYYY-mm-dd HH:MM. When using the --date option the period is all records after midnight at the start of YYYY-mm-dd up to and including midnight at the end of YYYY-mm-dd, in effect the same as --from=YYYY-mm-ddT00:00 --to=YYYY-mm-dd+1T00:00.

Note
--calc-missing uses the StdWXCalculate service to calculate missing derived observations. The data binding used by the StdWXCalculate service should normally match the data binding of the database being operated on by --calc-missing. User's who use custom or additional data bindings should take care to ensure the correct data bindings are used by both --calc-missing and the StdWXCalculate service. User's who use the default data binding need take no special precautions.

Action --check-strings

Normally, all entries in the archive are numbers. However, some SQLite database editors use a null string instead of a null value when deleting entries. These null strings can cause problems. This action checks the database to see if it contains any null strings.

wee_database --check-strings

Action --fix-strings

This action will check for any null strings in a SQLite database and if found substitute a true null value.

wee_database --fix-strings

wee_debug

Troubleshooting problems when running WeeWX often involves analysis of a number of pieces of seemingly disparate system and WeeWX related information. The wee_debug utility gathers all this information together into a single output to make troubleshooting easier. The wee_debug utility is particularly useful for new users as the output may be redirected to a file then emailed or posted to a forum to assist in remote troubleshooting.

Run the utility with the --help option to see how it is used:

wee_debug --help

This results in:

Usage: wee_debug --help
       wee_debug --info
            [CONFIG_FILE|--config=CONFIG_FILE]
            [--output|--output DEBUG_PATH]
            [--verbosity=0|1|2]
       wee_debug --version

Description:

Generate a standard suite of system/weewx information to aid in remote
debugging. The wee_debug output consists of two parts, the first part containing
a snapshot of relevant system/weewx information and the second part a parsed and
obfuscated copy of weewx.conf. This output can be redirected to file and posted
when seeking assistance via forums or email.

Actions:

--info           Generate a debug report.

Options:
  -h, --help            show this help message and exit
  --config=CONFIG_FILE  Use configuration file CONFIG_FILE.
  --info                Generate weewx debug output.
  --output              Write wee_debug output to DEBUG_PATH. DEBUG_PATH
                        includes path and file name. Default is
                        /var/tmp/weewx.debug.
  --verbosity=N         How much detail to display, 0-2, default=1.
  --version             Display wee_debug version number.

wee_debug will attempt to obfuscate obvious personal/private information in
weewx.conf such as user names, passwords and API keys; however, the user
should thoroughly check the generated output for personal/private information
before posting the information publicly.

Actions and options

Option --config

wee_debug /home/weewx/weewx.conf --info

or by using option --config:

wee_debug --config=/home/weewx/weewx.conf --info

Action --info

This action generates a debug report which can be sent off for remote debugging.

wee_debug --info

Warning!
The wee_debug output includes a copy of the in use WeeWX config file (usually weewx.conf) and whilst wee_debug attempts to obfuscate any personal or sensitive information in weewx.conf, the user should carefully check the wee_debug output for any remaining personal or sensitive information before emailing or posting the output publicly.

This results in output something like this:

Using verbosity=1, displaying most info

wee_debug output will be sent to stdout(console)

Using configuration file /home/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 'archive_mysql'

System info
  CPU implementer:        0x41
  Features:               half thumb fastmult vfp edsp java tls
  CPU architecture:       7
  BogoMIPS:               2.00
  Hardware:               BCM2708
  CPU revision:           7
  CPU part:               0xb76
  model name:             ARMv6-compatible processor rev 7 (v6l)
  Serial:                 000000009581b554
  processor:              0
  CPU variant:            0x0
  Revision:               000e

  Operating system:       debian 7.8
                          Linux rosella 4.1.6+ #810 PREEMPT Tue Aug 18 15:19:58 BST 2015 armv6l
  1 minute load average:  0.19
  5 minute load average:  0.15
  15 minute load average: 0.12

General weewx info
  Weewx version 3.2.1 detected.

Station info
  Station type: Simulator
  Driver:       weewx.drivers.simulator

Driver info
[Simulator]
    # This section is for the weewx weather station simulator

    # The time (in seconds) between LOOP packets.
    loop_interval = 2.5

    # The simulator mode can be either 'simulator' or 'generator'.
    # Real-time simulator. Sleep between each LOOP packet.
    mode = simulator
    # Generator.  Emit LOOP packets as fast as possible (useful for testing).
    #mode = generator

    # The start time. If not specified, the default is to use the present time.
    #start = 2011-01-01 00:00

    # The driver to use:
    driver = weewx.drivers.simulator

Currently installed extensions
Extension Name    Version   Description
Weewx-WD          1.2.0b1   Weewx support for Weather Display Live, SteelSeries Gauges and Carter Lake/Saratoga weather web site templates.

Archive info
  Database name:        weewx
  Table name:           archive
  Unit system:          16(METRIC)
  First good timestamp: 2013-01-01 00:00:00 AEST (1356962400)
  Last good timestamp:  2015-09-06 02:15:00 AEST (1441469700)
  Number of records:    281178
  weewx (weewx.conf) is set to use an archive interval of 300 seconds.
  The station hardware was not interrogated in determining archive interval.

Databases configured in weewx.conf
  Database name:        weewx
  Database driver:      weedb.mysql
  Database host:        localhost

  Database name:        wdsupp
  Database driver:      weedb.mysql
  Database host:        localhost

  Database name:        weewxwd
  Database driver:      weedb.mysql
  Database host:        localhost


Parsed and obfuscated weewx.conf
# WEEWX CONFIGURATION FILE
#
# Copyright (c) 2009-2015 Tom Keffer <tkeffer@gmail.com>
# See the file LICENSE.txt for your rights.

##############################################################################

# This section is for general configuration information.

... content removed for conciseness ...

#   This section configures the internal weewx engine.

[Engine]

    [[Services]]
        # This section specifies the services that should be run. They are
        # grouped by type, and the order of services within each group
        # determines the order in which the services will be run.
        prep_services = weewx.engine.StdTimeSynch
        data_services = ,
        process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate, user.weewxwd3.WdWXCalculate
        archive_services = weewx.engine.StdArchive, user.weewxwd3.WdArchive, user.weewxwd3.WdSuppArchive
        restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS, user.sync.SyncService
        report_services = weewx.engine.StdPrint, weewx.engine.StdReport

################################################################################

wee_debug report successfully generated

Option --output

By default, wee_debug sends its output to the system "standard output" (stdout) unless the --output option is used.

The option --output with no parameter sends output to the default file /var/tmp/weewx.debug. Example:

wee_debug --info --output

The option --output with a specified file will send it to that file. Example:

wee_debug --info --output /home/weewx/another.debug

Option --verbosity

The amount of information included in the wee_debug output can be changed using the --verbosity option. The --verbosity option can be set to 0, 1 or 2 with each higher level successively displaying more information. The default level is 1. The information displayed for each level is:

Level	Included Information
--verbosity 0	path and name of WeeWX config file used (usually weewx.conf)
	name of WeeWX database binding used
	operating system version
	WeeWX version number
	WeeWX station type and driver name
	summary of currently installed extensions
	summary of WeeWX archive
	parsed and obfuscated WeeWX config file (usually weewx.conf)
--verbosity 1	as per --verbosity 0
	cpu info summary
	system load averages
	driver config extract from WeeWX config file (usually weewx.conf)
	summary of databases configured in WeeWX config file (usually weewx.conf)
--verbosity 2	as per --verbosity 1
	list of supported SQL keys
	list of supported observation types

wee_device

The utility wee_device is used to configure hardware settings, such as rain bucket size, station archive interval, altitude, EEPROM constants, etc., on your station. In order to do its job, it depends on optional code being present in the hardware driver. Because not all drivers have this code, it may not work for your specific device. If it does not, you will have to consult your manufacturer's instructions for how to set these things through your console or other means.

wee_device uses the option station_type in weewx.conf to determine what device you are using and what options to display. Make sure it's set correctly before attempting to use this utility.

Because wee_device uses hardware-specific code, its options are different for every station type. You should run it with --help to see how to use it for your specific station:

wee_device --help

The utility requires a weewx.conf file. If no file is specified, it will look for weewx.conf in the standard location. If your configuration file is in a non-standard location, specify the path to the configuration file as the first argument. For example,

wee_device /path/to/weewx.conf --help

For details about the options available for each type of hardware, see the appropriate hardware section:

wee_extension

The utility wee_extension is used to add and remove extensions. Use the --help option to see how it is used:

wee_extension --help

This results in:

Usage: wee_extension --help
       wee_extension --list
           [CONFIG_FILE|--config=CONFIG_FILE]
       wee_extension --install=(filename|directory)
           [CONFIG_FILE|--config=CONFIG_FILE]
           [--tmpdir==DIR] [--dry-run] [--verbosity=N]
       wee_extension --uninstall=EXTENSION
           [CONFIG_FILE|--config=CONFIG_FILE]
           [--verbosity=N]

Install, list, and uninstall extensions to weewx.

Actions:

--list:      Show installed extensions.
--install:   Install the specified extension.
--uninstall: Uninstall the specified extension.

Options:
  -h, --help            show this help message and exit
  --list                Show installed extensions.
  --install=FILENAME|DIRECTORY
                        Install an extension contained in FILENAME  (such as
                        pmon.tar.gz), or from a DIRECTORY in which it has been
                        unpacked.
  --uninstall=EXTENSION
                        Uninstall the extension with name EXTENSION.
  --config=CONFIG_FILE  Use configuration file CONFIG_FILE.
  --tmpdir=DIR          Use temporary directory DIR.
  --bin-root=BIN_ROOT   Look in BIN_ROOT for weewx executables.
  --dry-run             Print what would happen but do not do it.
  --verbosity=N         How much status to display, 0-3

Actions and options

Action --install

Use this action to install an extension. You must specify the path to a .zip archive, a .tgz/.tar.gz archive, or a directory. If a directory, the extension must have been unpacked into it.

wee_extension --install=examples/basic
wee_extension --install=basic.tar.gz

Action --list

This action will list all the extensions that you have installed.

wee_extension --list

Action --uninstall

Use this action to remove an extension. You must specify the name of the extension, without any version number or zip/tgz extension.

wee_extension --uninstall=basic

Example: install an extension

The wee_extension utility makes a copy of any file or directory that it modifies or replaces. When installing, it creates a directory called installer in the user directory. The contents of the installer directory are used to enumerate and uninstall extensions.

Let's try installing a simple extension, cmon, used to monitor your computer.

First download it. You can either do this from the link given in the wiki, or by using wget (which you may have to install):

wget -P /var/tmp http://lancet.mit.edu/mwall/projects/weather/releases/weewx-cmon-0.7.tgz

This will put the tarfile weewx-cmon-0.7.tgz in the directory /var/tmp.

Now install the extension:

wee_extension --install=/var/tmp/weewx-cmon-0.7.tgz
Request to install '/var/tmp/weewx-cmon-0.7.tgz'
Extracting from tarball /var/tmp/weewx-cmon-0.7.tgz
Added new service user.cmon.ComputerMonitor to process_services
Saving installer file to /home/weewx/bin/user/installer/cmon
Saved configuration dictionary. Backup copy at /home/weewx/weewx.conf.20150430130322
Finished installing extension '/var/tmp/weewx-cmon-0.7.tgz'

The installer has done a number of things for you:

It put a new skin, cmon, in the skins subdirectory;
It put a new service, user.cmon.ComputerMonitor, in the list of services to be run by WeeWX;
It defined a new database, cmon_sqlite, and a binding, cmon_binding, to that database;
It added a top-level "stanza" [ComputerMonitor] to your configuration file weewx.conf, that specifies the data binding cmon is to use.
And, finally, it saved the details of how the extension was installed so you can remove it later, should you choose to do so.

Now you can use the --list action to see which extensions are installed:

wee_extension --list
Extension Name   Version  Description
cmon             0.7      Collect and display computer health indicators

You can see that it listed the extension we just installed, cmon.

You can remove an extension using the --uninstall action:

wee_extension --uninstall=cmon
wee_extension --list
No extensions installed

wee_import

Some WeeWX users will have historical data from another source (e.g., other weather station software or a manually compiled file) which they wish to import into WeeWX. Such data can, depending upon the source, be imported using the wee_import utility. This section details the use of the wee_import utility.

The wee_import utility supports importing observational data from the following sources:

a single Comma Separated Values (CSV) format file
the historical observations of a Weather Underground personal weather station
one or more Cumulus monthly log files
one or more Weather Display monthly log files

Before starting, it's worth running the utility with the --help flag to see how wee_import is used:

wee_import --help

This will result in an output that looks something like this:

Usage: wee_import --help
       wee_import --version
       wee_import --import-config=IMPORT_CONFIG_FILE
            [--config=CONFIG_FILE]
            [--date=YYYY-mm-dd | --from=YYYY-mm-dd[THH:MM] --to=YYYY-mm-dd[THH:MM]]
            [--dry-run]
            [--verbose]
            [--no-prompt]
            [--suppress-warnings]


Import observation data into a WeeWX archive.

Options:
  -h, --help            show this help message and exit
  --config=CONFIG_FILE  Use configuration file CONFIG_FILE.
  --import-config=IMPORT_CONFIG_FILE
                        Use import configuration file IMPORT_CONFIG_FILE.
  --dry-run             Print what would happen but do not do it.
  --date=YYYY-mm-dd     Import data for this date. Format is YYYY-mm-dd.
  --from=YYYY-mm-dd[THH:MM]
                        Import data starting at this date or date-time. Format
                        is YYYY-mm-dd[THH:MM].
  --to=YYYY-mm-dd[THH:MM]
                        Import data up until this date or date-time. Format is
                        YYYY-mm-dd[THH:MM].
  --verbose             Print and log useful extra output.
  --no-prompt           Do not prompt. Accept relevant defaults and all y/n
                        prompts.
  --suppress-warnings   Suppress warnings to stdout. Warnings are still
                        logged.
  --version             Display wee_import version number.

wee_import will import data from an external source into a WeeWX
archive. Daily summaries are updated as each archive record is
imported so there should be no need to separately rebuild the daily
summaries using the wee_database utility.

Actions and options

The wee_import actions and options are described in more detail below:

Option --config

The utility is pretty good at "guessing" where your configuration file weewx.conf is, but if you've done an unusual install, you may have to tell it explicitly. You can do this by using the --config option:

wee_import --config=/this/directory/weewx.conf --import-config=/directory/import.conf

Option --import-config

wee_import uses a secondary configuration file to store various import parameters. The --import-config option is mandatory for all imports. Example import configuration files for each type of import supported by wee_import are provided in the util/import directory. These example files are best used by making a copy of the applicable example file in a working directory and then modifying the duplicate file to suit your needs. The --import-config option is used as follows:

wee_import --import-config=/directory/import.conf

Option --dry-run

The inclusion of the --dry-run option will cause the import to proceed but no actual data will be saved to the database. This is a useful option to use when first importing data.

wee_import --import-config=/directory/import.conf --dry-run

Option --date

Records from a single date can be imported by use of the --date option. The --date option accepts strings of the format YYYY-mm-dd. Whilst the use of the --date option will limit the imported data to that of a single date, the default action if the --date option (and the --from and --to options) is omitted may vary depending on the source. The operation of the --date option is summarised in the following table:

Option --date
option	Records imported for a CSV, Cumulus, Weather Display or WeatherCat import	Records imported for a Weather Underground import
omitted (i.e., the default)	All available records	Todays records only
--date=2015-12-22	All records from 2015-12-22 00:00 (exclusive) to 2015-12-23 00:00 (inclusive)	All records from 2015-12-22 00:00 (exclusive) to 2015-12-23 00:00 (inclusive)

Note
If the --date, --from and --to options are omitted the default is to import today's records only when importing from Weather Underground or to import all available records when importing from any other source.

Note
WeeWX considers an archive record to represent an aggregation of data over the archive interval preceding the archive record's timestamp. For this reason imports which are to be limited to a given date with the --date option will only import records timestamped after midnight at the start of the day concerned and up to and including midnight at the end of the day concerned.

Options --from and --to

Whilst the --date option allows imported data to be limited to a single date, the --from and --to options allow finer control by importing only the records that fall within the date or date-time range specified by the --from and --to options. The --from option determines the earliest (inclusive), and the --to option determines the latest (exclusive), date or date-time of the records being imported. The --from and --to options accept a string of the format YYYY-mm-dd[THH:MM]. The T literal is mandatory if specifying a date-time.

Note
The --from and --to options must be used as a pair, they cannot be used individually or in conjunction with the --date option.

The operation of the --from and --to options is summarised in the following table:

Options --from and --to
options		Records imported for a CSV, Cumulus, Weather Display or WeatherCat import	Records imported for a Weather Underground import
omitted (i.e., the default)	omitted (i.e., the default)	All available records	Todays records only
--from=2015-12-22	--to=2015-12-29	All records from 2015-12-22 00:00 (exclusive) to 2015-12-30 00:00 (inclusive)	All records from 2015-12-22 00:00 (exclusive) to 2015-12-30 00:00 (inclusive)
--from=2016-7-18T15:29	--to=2016-7-25	All records from 2016-7-18 15:29 (exclusive) to 2016-7-26 00:00 (inclusive)	All records from 2016-7-18 15:29 (exclusive) to 2016-7-26 00:00 (inclusive)
--from=2016-5-12	--to=2016-7-22T22:15	All records from 2016-5-12 00:00 (exclusive) to 2016-7-22 22:15 (inclusive)	All records from 2016-5-12 00:00 (exclusive) to 2016-7-22 22:15 (inclusive)
--from=2016-3-18T15:29	--to=2016-6-20T22:00	All records from 2016-3-18 15:29 (exclusive) to 2016-6-20 22:00 (inclusive)	All records from 2016-3-18 15:29 (exclusive) to 2016-6-20 22:00 (inclusive)

Note
WeeWX considers an archive record to represent an aggregation of data over the archive interval preceding the archive record's timestamp. For this reason imports which are to be limited to a given timespan with the --from and --to options will only import records timestamped after the timestamp represented by the --from option and up to and including the timestamp represented by the --to option.

Option --verbose

Inclusion of the --verbose option will cause additional information to be printed during wee_import execution.

wee_import --import-config=/directory/import.conf --verbose

Option --no-prompt

Inclusion of the --no-prompt option will run wee_import without prompts. Relevant defaults will be used and all y/n prompts are automatically accepted as 'y'. This may be useful for unattended use of wee_import.

wee_import --import-config=/directory/import.conf --no-prompt

Warning!
Care must be taken when using the --no-prompt option as ignoring warnings during the import process can lead to unexpected results. Whilst existing data will be protected, the use or acceptance of an incorrect or unexpected parameter or default may lead to significant amounts of unwanted data being imported.

Option --suppress-warnings

The --suppress-warnings option suppresses wee_import warning messages from being displayed on the console during the import. wee_import may issue a number of warnings during import. These warnings may be due to the source containing more than one entry for a given timestamp or there being no data found for a mapped import field. These warnings do not necessarily require action, but they can consist of extensive output and thus make it difficult to follow the import progress. Irrespective of whether --suppress-warnings is used all warnings are sent to log.

wee_import --import-config=/directory/import.conf --suppress-warnings

The import configuration file

wee_import requires a second configuration file, the import configuration file, in addition to the standard WeeWX configuration file. The import configuration file specifies the import type and various options associated with each type of import. The import configuration file is specified using the mandatory --import-config option. How you construct the import configuration file is up to you; however, the recommended method is to copy one of the example import configuration files located in the util/import directory, modify the configuration options in the newly copied file to suit the import to be performed and then use this file as the import configuration file.

Following is the definitive guide to the options available in the import configuration file. Default values are provided for a number of options, meaning that if they are not listed in the import configuration file at all wee_import will pick sensible values. When the documentation below gives a "default value" this is the value that will be used if the option is omitted.

General

source

The source option determines the type of import to be performed by wee_import. The option is mandatory and must be set to one of the following:

CSV to import from a single CSV format file.
WU to import from a Weather Underground PWS history.
Cumulus to import from one or more Cumulus monthly log files.
WD to import from one or more Weather Display monthly log files.
WeatherCat to import from one or more WeatherCat monthly .cat files.

There is no default.

[CSV]

The [CSV] section contains the options relating to the import of observational data from a CSV format file.

file

The file containing the CSV format data to be used as the source during the import. Include full path and filename. There is no default.

source_encoding

The source file encoding. This parameter is optional and should only need be used if the source file uses an encoding other than UTF-8 or an ASCII compatible encoding. If used, the setting used should be a Python Standard Encoding. The default value is utf-8-sig.

delimiter

The one character string used to separate fields. Default is , (comma).

interval

Determines how the time interval (WeeWX archive table field interval) between successive observations is derived. The interval can be derived by one of three methods:

The interval can be calculated as the time, rounded to the nearest minute, between the date-time of successive records. This method is suitable when the data was recorded at fixed intervals and there are NO missing records in the source data. Use of this method when there are missing records in the source data can compromise the integrity of the WeeWX statistical data. Select this method by setting interval = derive.
The interval can be set to the same value as the archive_interval setting under [StdArchive] in weewx.conf. This setting is useful if the data was recorded at fixed intervals but there are some missing records and the fixed interval is the same as the archive_interval setting under [StdArchive] in weewx.conf. Select this method by setting interval = conf.
The interval can be set to a fixed number of minutes. This setting is useful if the source data was recorded at fixed intervals but there are some missing records and the fixed interval is different to the archive_interval setting under [StdArchive] in weewx.conf. Select this method by setting interval = x where x is an integer number of minutes.

The default value is derive. If the CSV source data records are equally spaced in time, but some records are missing, then a better result may be achieved using conf or a fixed interval setting.

qc

Determines whether simple quality control checks are applied to imported data. Setting qc = True will result in wee_import applying the WeeWX StdQC minimum and maximum checks to any imported observations. wee_import quality control checks use the same configuration settings, and operate in the same manner, as the StdQC service. For example, for minimum/maximum quality checks, if an observation falls outside of the quality control range for that observation, then the observation will be set to None. In such cases you will be alerted through a short message similar to:

2016-01-12 10:00:00 AEST (1452556800) record value 'outTemp' 194.34 outside limits (0.0, 120.0)

As derived observations are calculated after the quality control check is applied, derived observations are not subject to quality control checks. Setting qc = False will result in wee_import not applying quality control checks to imported data. The default is True.

calc_missing

Determines whether any missing derived observations will be calculated from the imported data. Setting calc_missing = True will result in wee_import using the WeeWX StdWXCalculate service to calculate any missing derived observations from the imported data. Setting calc_missing = False will result in WeeWX leaving any missing derived observations as None. The observations that StdWXCalculate can calculate are listed in the [StdWXCalculate] section of the User's Guide. The default is True.

ignore_invalid_data

Determines whether invalid data in a source field is ignored or the import aborted. If invalid data is found in a source field and ignore_invalid_data is True the corresponding WeeWX destination field is set to None and the import continues. If invalid data is found in a source field and ignore_invalid_data is False the import is aborted. The default is True.

tranche

To speed up database operations imported records are committed to database in groups of records rather than individually. The size of the group is set by the tranche parameter. Increasing the tranche parameter may result in a slight speed increase but at the expense of increased memory usage. Decreasing the tranche parameter will result in less memory usage but at the expense of more frequent database access and likely increased time to import. The default is 250 which should suit most users.

UV_sensor

WeeWX records a None/null for UV when no UV sensor is installed, whereas some weather station software records a value of 0 for UV index when there is no UV sensor installed. The UV_sensor parameter enables wee_import to distinguish between the case where a UV sensor is present and the UV index is 0 and the case where no UV sensor is present and UV index is 0. UV_sensor = False should be used when no UV sensor was used in producing the source data. UV_sensor = False will result in None/null being recorded in the WeeWX archive field UV irrespective of any UV observations in the source data. UV_sensor = True should be used when a UV sensor was used in producing the source data. UV_sensor = True will result in UV observations in the source data being stored in the WeeWX archive field UV. The default is True.

solar_sensor

WeeWX records a None/null when no solar radiation sensor is installed, whereas some weather station software records a value of 0 for solar radiation when there is no solar radiation sensor installed. The solar_sensor parameter enables wee_import to distinguish between the case where a solar radiation sensor is present and solar radiation is 0 and the case where no solar radiation sensor is present and solar radiation is 0. solar_sensor = False should be used when no solar radiation sensor was used in producing the source data. solar_sensor = False will result in None/null being recorded in the WeeWX archive field radiation irrespective of any solar radiation observations in the source data. solar_sensor = True should be used when a solar radiation sensor was used in producing the source data. solar_sensor = True will result in solar radiation observations in the source data being stored in the WeeWX archive field radiation. The default is True.

raw_datetime_format

WeeWX records each record with a unique unix epoch timestamp, whereas many weather station applications or web sources export observational data with a human readable date-time. This human readable date-time is interpreted according to the format set by the raw_datetime_format option. This option consists of Python strptime() format codes and literal characters to represent the date-time data being imported. For example, if the source data uses the format 23 January 2015 15:34 then the appropriate setting for raw_datetime_format would be %d %B %Y %H:%M, 9:25:00 12/28/16 would use %H:%M:%S %m/%d/%y. If the source data provides a unix epoch timestamp as the date-time field then the unix epoch timestamp is used directly and the raw_datetime_format option is ignored. The default is %Y-%m-%d %H:%M:%S.

Note
wee_import does not support the construction of the unique record date-time stamp from separate date and time fields, rather the date-time information for each imported record must be contained in a single field. CSV data containing separate date and time fields may require further manual processing before they can be imported.

rain

The WeeWX rain field records rainfall that was recorded in the preceding archive period, so for a five minute archive period the rain field in each archive record would contain the total rainfall that fell in the previous five minutes. Many weather station applications provide a daily or yearly total. wee_import can derive the WeeWX rain field in one of two ways:

If the imported rainfall data is a running total then wee_import can derive the WeeWX rain field from successive totals. For this method use rain = cumulative.
If the imported rainfall data is a discrete value per date-time period then rain = discrete should be used.

Note
wee_import only supports cumulative rainfall data that resets on a midnight boundary, cumulative rainfall data that resets at some other time; e.g., 9am, is not supported. In such cases the rainfall data will need to be converted to either reset on a midnight boundary or a discrete value per date-time period and rain = discrete used. The former may be possible by selecting another rainfall field (if available) in the source data, otherwise it will require manual manipulation of the source data.

wind_direction

WeeWX records wind direction in degrees as a number from 0 to 360 inclusive (no wind direction is recorded as None/null), whereas some data sources may provide wind direction as number over a different range (e.g., -180 to +180) or may use a particular value when there is no wind direction (e.g., 0 may represent no wind direction and 360 may represent a northerly wind, or -9999 (or some similar clearly invalid number) to represent there being no wind direction). wee_import handles such variations in data by defining a range over which imported wind direction values are accepted. Any value outside of this range is treated as there being no wind direction and is recorded as None/null. Any value inside the range is normalised to the range 0 to 360 inclusive (e.g., -180 would be normalised to 180). The wind_direction option consists of two comma separated numbers of the format lower, upper where lower and upper are inclusive. The operation of the wind_direction option is best illustrated through the following table:

Option wind_direction
wind_direction option setting	Source data wind direction value	Imported wind direction value
0, 360	0	0
	160	160
	360	360
	500	None/null
	-45	None/null
	-9999	None/null
	No data	None/null
-360, 360	0	0
	160	160
	360	360
	500	None/null
	-45	315
	-9999	None/null
	No data	None/null
-180, 180	0	0
	160	160
	360	None/null
	500	None/null
	-45	315
	-9999	None/null
	No data	None/null

The default is 0, 360.

[[FieldMap]]

The [[FieldMap]] stanza defines the mapping from the source data fields to WeeWX archive fields. The map consists of one row per field using the format:

weewx_archive_field_name = csv_field_name, weewx_unit_name

Where weewx_archive_field_name is a field name in the in-use WeeWX archive table schema, csv_field_name is the name of a field from the CSV file and weewx_unit_name is the WeeWX unit name of the units used by csv_field_name. This mapping allows wee_import to take a source data field, do the appropriate unit conversion and store the resulting value in the appropriate WeeWX archive field. A mapping is not required for every WeeWX archive field (e.g., the source may not provide inside temperature so no inTemp field mapping is required) and neither does every CSV field need to be included in a mapping (e.g., the source data field monthrain may have no use if the source data field dayrain provides the data for the WeeWX archive rain field). Unused field mapping lines will not be used and may be omitted.

If the source data includes a field that contains a WeeWX unit system code (i.e. the equivalent of the WeeWX usUnits field such as may be obtained from WeeWX or wview data) then this field can be mapped to the WeeWX usUnits field and used to set the units used for all fields being imported. In such cases the the weewx_unit_name portion of the imported fields in the field map is not used and can be omitted.

For example, source CSV data with the following structure:

date_and_time,temp,humid,wind,dir,rainfall,rad,river
23 May 2018 13:00,17.4,56,3.0,45,0.0,956,340
23 May 2018 13:05,17.6,56,1.0,22.5,0.4,746,341

where temp is temperature in Celsius, humid is humidity in percent, wind is wind speed in km/h, dir is wind direction in degrees, rainfall is rain in mm, rad is radiation in watts per square meter and river is river height in mm might use a field map as follows:

[[FieldMap]]
    dateTime    = date_and_time, unix_epoch
    outTemp     = temp, degree_C
    outHumidity = humid, percent
    windSpeed   = wind, km_per_hour
    windDir     = dir, degree_compass
    rain        = rainfall, mm
    radiation   = rad, watt_per_meter_squared

If the same source CSV data included a field unit_info that contains WeeWX unit system data as follows:

date_and_time,temp,humid,wind,dir,rainfall,rad,river,unit_info
23 May 2018 13:00,17.4,56,3.0,45,0.0,956,340,1
23 May 2018 13:05,17.6,56,1.0,22.5,0.4,746,341,16

then a field map such as the following might be used:

[[FieldMap]]
    dateTime    = date_and_time, unix_epoch
    usUnits     = unit_info
    outTemp     = temp
    outHumidity = humid
    windSpeed   = wind
    windDir     = dir
    rain        = rainfall
    radiation   = rad

Note
Any WeeWX archive fields that are derived (e.g., dewpoint) and for which there is no field mapping may be calculated during import by use of the calc_missing option in the [CSV] section of the import configuration file.

[WU]

The [WU] section contains the options relating to the import of observational data from a Weather Underground PWS history.

station_id

The Weather Underground weather station ID of the PWS from which the historical data will be imported. There is no default.

api_key

The Weather Underground API key to be used to obtain the PWS history data. There is no default.

Note
The API key is a seemingly random string of 32 characters used to access the new (2019) Weather Underground API. PWS contributors can obtain an API key by logging onto the Weather Underground internet site and accessing Member Settings. 16 character API keys used with the previous Weather Underground API are not supported.

interval

Determines how the time interval (WeeWX database field interval) between successive observations is determined. This option is identical in operation to the CSV interval option but applies to Weather Underground imports only. As a Weather Underground PWS history sometimes has missing records, the use of interval = derive may give incorrect or inconsistent interval values. Better results may be obtained by using interval = conf if the current WeeWX installation has the same archive_interval as the Weather Underground data, or by using interval = x where x is the time interval in minutes used to upload the Weather Underground data. The most appropriate setting will depend on the completeness and (time) accuracy of the Weather Underground data being imported.

The default is derive.

qc

Determines whether simple quality control checks are applied to imported data. This option is identical in operation to the CSV qc option but applies to Weather Underground imports only. As Weather Underground imports at times contain nonsense values, particularly for fields for which no data was uploaded to Weather Underground by the PWS, the use of quality control checks on imported data can prevent these nonsense values from being imported and contaminating the WeeWX database. The default is True.

calc_missing

Determines whether any missing derived observations will be calculated from the imported data. This option is identical in operation to the CSV calc_missing option but applies to Weather Underground imports only. The default is True.

ignore_invalid_data

Determines whether invalid data in a source field is ignored or the import aborted. This option is identical in operation to the CSV ignore_invalid_data option but applies to Weather Underground imports only. The default is True.

tranche

The number of records written to the WeeWX database in each transaction. This option is identical in operation to the CSV tranche option but applies to Weather Underground imports only. The default is 250 which should suit most users.

wind_direction

Determines the range of acceptable wind direction values in degrees. This option is identical in operation to the CSV wind_direction option but applies to Weather Underground imports only. The default is 0, 360 which should suit most users.

[Cumulus]

The [Cumulus] section contains the options relating to the import of observational data from Cumulus monthly log files.

source_encoding

The Cumulus monthly log file encoding. This option is identical in operation to the CSV source_encoding option but applies to Cumulus imports only. The default is utf-8-sig.

interval

Determines how the time interval (WeeWX database field interval) between successive observations is determined. This option is identical in operation to the CSV interval option but applies to Cumulus monthly log file imports only. As Cumulus monthly log files can, at times, have missing entries, the use of interval = derive may give incorrect or inconsistent interval values. Better results may be obtained by using interval = conf if the archive_interval for the current WeeWX installation is the same as the Cumulus 'data log interval' setting used to generate the Cumulus monthly log files, or by using interval = x where x is the time interval in minutes used as the Cumulus 'data log interval' setting. The most appropriate setting will depend on the completeness and (time) accuracy of the Cumulus data being imported.

The default is derive.

qc

Determines whether simple quality control checks are applied to imported data. This option is identical in operation to the CSV qc option but applies to Cumulus imports only. The default is True.

calc_missing

separator

The character used as the date field separator in the Cumulus monthly log file. A solidus (/) is frequently used but it may be another character depending on the settings on the machine that produced the Cumulus monthly log files. This parameter must be included in quotation marks. Default is '/'.

delimiter

The character used as the field delimiter in the Cumulus monthly log file. A comma is frequently used but it may be another character depending on the settings on the machine that produced the Cumulus monthly log files. This parameter must be included in quotation marks. Default is ','.

decimal

The character used as the decimal point in the Cumulus monthly log files. A full stop is frequently used but it may be another character depending on the settings on the machine that produced the Cumulus monthly log files. This parameter must be included in quotation marks. Default is '.'.

ignore_invalid_data

Determines whether invalid data in a source field is ignored or the import aborted. This option is identical in operation to the CSV ignore_invalid_data option but applies to Cumulus monthly log file imports only. The default is True.

tranche

The number of records written to the WeeWX database in each transaction. This option is identical in operation to the CSV tranche option but applies to Cumulus monthly log file imports only. The default is 250 which should suit most users.

UV_sensor

Enables wee_import to distinguish between the case where a UV sensor is present and the UV index is 0 and the case where no UV sensor is present and UV index is 0. This option is identical in operation to the CSV UV_sensor option but applies to Cumulus monthly log file imports only. The default is True.

solar_sensor

Enables wee_import to distinguish between the case where a solar radiation sensor is present and the solar radiation is 0 and the case where no solar radiation sensor is present and solar radiation is 0. This option is identical in operation to the CSV solar_sensor option but applies to Cumulus monthly log file imports only. The default is True.

[[Units]]

The [[Units]] stanza defines the units used in the Cumulus monthly log files. Units settings are required for temperature, pressure, rain and speed. The format for each setting is:

obs_type = weewx_unit_name

Where obs_type is one of temperature, pressure, rain or speed and weewx_unit_name is the WeeWX unit name of the units used by that particular obs_type. As Cumulus supports a different suite of possible units only a subset of the available WeeWX unit names can be used for some settings.

[WD]

The [WD] section contains the options relating to the import of observational data from Weather Display monthly log files.

logs_to_process

The Weather Display monthly log files to be processed. Weather Display uses multiple files to record each month of data. Which monthly log files are produced depends on the Weather Display configuration and the capabilities of the weather station. wee_import supports the following Weather Display monthly log files:

MMYYYYlg.txt
MMYYYYlgcsv.csv (csv format version of MMYYYYlg.txt)
MMYYYYvantagelog.txt
MMYYYYvantagelogcsv.csv (csv format version of MMYYYYvantagelog.txt)
MMYYYYvantageextrasensorslog.csv

where MM is a one or two digit month and YYYY is a four digit year

The format for the logs_to_process setting is:

logs_to_process = [lg.txt, | logcsv.csv, | vantagelog.txt, | vantagelogcsv.csv, | vantageextrasensorslog.csv]

Note
The leading MMYYYY is omitted when listing the monthly log files to be processed using the logs_to_process setting. Inclusion of the leading MMYYYY will cause the import to fail.

Note
The MMYYYYlgcsv.csv and MMYYYYvantagelogcsv.csv log files are CSV versions of MMYYYYlg.txt and MMYYYYvantagelog.txt respectively. Either the .txt or .csv version of these files should be used but not both.

The monthly log files selected for processing should be chosen carefully as the selected log files will determine the Weather Display data fields available for import. wee_import is able to import the following data from the indicated monthly log files:

MMYYYYlg.txt/MMYYlgcsv.csv:

average wind speed
barometer
date and time
dew point
heat index
outside humidity
outside temperature
rain fall
wind direction
wind gust speed

MMYYYYvantagelog.txt/MMYYYYvantagelogcsv.csv:

date and time
soil moisture
soil temperature
solar radiation
UV index

MMYYYYvantageextrasensorslog.csv:

date and time
extra humidity 1
extra humidity 2
extra humidity 3
extra humidity 4
extra humidity 5
extra humidity 6
extra temperature 1
extra temperature 2
extra temperature 3
extra temperature 4
extra temperature 5
extra temperature 6

Note
Whilst the above log files may contain the indicated data the data may only be imported subject to a suitable field map and in-use WeeWX archive table schema (refer to the [[FieldMap]] option).

The default is lg.txt, vantagelog.txt, vantageextrasensorslog.csv.

source_encoding

The Weather Display monthly log file encoding. This option is identical in operation to the CSV source_encoding option but applies to Weather Display imports only. The default is utf-8-sig.

interval

Determines how the time interval (WeeWX database field interval) between successive observations is determined. This option is identical in operation to the CSV interval option but applies to Weather Display monthly log file imports only. As Weather Display log files nominally have entries at one minute intervals the recommended approach is to set interval = 1. As Weather Display monthly log files can, at times, have missing entries, the use of interval = derive may give incorrect or inconsistent interval values. If the archive_interval for the current WeeWX installation is 1 minute interval = conf may be used. In most cases the most appropriate setting will be interval = 1.

The default is 1.

qc

Determines whether simple quality control checks are applied to imported data. This option is identical in operation to the CSV qc option but applies to Weather Display imports only. The default is True.

calc_missing

txt_delimiter

The character used as the field delimiter in Weather Display text format monthly log files (.txt files). A space is normally used but another character may be used if necessary. This parameter must be included in quotation marks. Default is ' '.

csv_delimiter

The character used as the field delimiter in Weather Display csv format monthly log files (.csv files). A comma is normally used but another character may be used if necessary. This parameter must be included in quotation marks. Default is ','.

decimal

The character used as the decimal point in the Weather Display monthly log files. A full stop is frequently used but another character may be used if necessary. This parameter must be included in quotation marks. Default is '.'.

ignore_Missing_log

Determines whether missing log files are to be ignored or the import aborted. Weather Display log files are complete in themselves and a missing log file will have no effect other than there will be no imported data for the period covered by the missing log file. The default is True.

ignore_invalid_data

tranche

The number of records written to the WeeWX database in each transaction. This option is identical in operation to the CSV tranche option but applies to Weather Display monthly log file imports only. The default is 250 which should suit most users.

UV_sensor

Enables wee_import to distinguish between the case where a UV sensor is present and the UV index is 0 and the case where no UV sensor is present and UV index is 0. This option is identical in operation to the CSV UV_sensor option but applies to Weather Display monthly log file imports only. The default is True.

solar_sensor

Enables wee_import to distinguish between the case where a solar radiation sensor is present and the solar radiation is 0 and the case where no solar radiation sensor is present and solar radiation is 0. This option is identical in operation to the CSV solar_sensor option but applies to Weather Display monthly log file imports only. The default is True.

ignore_extreme_temp_hum

Determines whether extreme temperature and humidity values are ignored. Weather Display log files record the value 255 for temperature and humidity fields if no corresponding sensor is present. Setting ignore_extreme_temp_hum = True will cause temperature and humidity values of 255 to be ignored. Setting ignore_extreme_temp_hum = False will cause temperature and humidity values of 255 to be treated as valid data to be imported. The default is True.

Note
Setting ignore_extreme_temp_hum = False will cause temperature and humidity values of 255 to be imported; however, these values may be rejected by the simple quality control checks implemented if qc = True is used.

[[Units]]

The [[Units]] stanza defines the units used in the Weather Display monthly log files. Weather Display monthly log files normally use Metric or US customary units depending on the Log File setting under Units on the Units/Wind Chill tab of the Weather Display Universal Setup. In such cases the units configuration option may be set to Metric or US to select either Metric or US customary units. There is no default.

It is also possible to individually specify the log file units used for temperature, pressure, rain and speed. The format for each setting is:

obs_type = weewx_unit_name

Where obs_type is one of temperature, pressure, rain or speed and weewx_unit_name is the WeeWX unit name of the units used by that particular obs_type. As Weather Display supports a different suite of possible units only a subset of the available WeeWX unit names can be used for some settings.

The preferred method for defining the Weather Display log file units is through the use of the units configuration option. When defining the import log file units either the units configuration option should be used or the individual temperature, pressure, rain and speed units defined but not both. If both the units configuration option is defined as well as the individual temperature, pressure, rain and speed units defined then the units configuration option takes precedence and all other units settings are ignored.

[[FieldMap]]

The [[FieldMap]] stanza defines the mapping from the Weather Display monthly log data fields to WeeWX archive fields. By default imported Weather Display data is mapped to the corresponding WeeWX archive fields using a default field map. The default field map will likely suit most users; however, depending on the station capabilities and the in-use WeeWX database schema, a custom field map may be required if Weather Display monthly logs contain data from additional sensors that cannot be stored in the WeeWX archive using the default field map. A custom field map also makes it possible to limit the Weather Display monthly log data fields that are imported into WeeWX.

The field map consists of one row per field using the format:

weewx_archive_field_name = weather_display_field_name

Where weewx_archive_field_name is a field name in the in-use WeeWX archive table schema and weather_display_field_name is a Weather Display import field name. The available Weather Display import field names are listed in the table below.

Available Weather Display import fields
Field name	Description
barometer	barometric pressure
dewpoint	dew point
direction	wind direction
gustspeed	wind gust speed
heatindex	heat index
humidity	outside humidity
hum1	extra humidity 1
hum2	extra humidity 2
hum3	extra humidity 3
hum4	extra humidity 4
hum5	extra humidity 5
hum6	extra humidity 6
radiation	solar radiation
rainlastmin	rainfall in the last 1 minute
soilmoist	soil moisture
soiltemp	soil temperature
temperature	outside temperature
temp1	extra temperature 1
temp2	extra temperature 2
temp3	extra temperature 3
temp4	extra temperature 4
temp5	extra temperature 5
temp6	extra temperature 6
uv	UV index
windspeed	average wind speed

A mapping is not required for every WeeWX archive field (e.g., the Weather Display monthly logs may not provide inside temperature so no inTemp field mapping is required) and neither does every Weather Display monthly log field need to be included in a mapping (e.g., the Weather Display monthly log field soiltemp may have no data as the station has no soil temperature probe).

The example Weather Display import configuration file located in the util/import directory contains an example field map in the import configuration file comments. There is no default.

[WeatherCat]

The [WeatherCat] section contains the options relating to the import of observational data from WeatherCat monthly .cat files.

source_encoding

The WeatherCat monthly .cat file encoding. This option is identical in operation to the CSV source_encoding option but applies to WeatherCat imports only. The default is utf-8-sig.

interval

Determines how the time interval (WeeWX database field interval) between successive observations is determined. This option is identical in operation to the CSV interval option but applies to WeatherCat imports only. As WeatherCat monthly .cat files can, at times, have missing entries, the use of interval = derive may give incorrect or inconsistent interval values. Better results may be obtained by using interval = conf if the archive_interval for the current WeeWX installation is the same as the WeatherCat .cat file log interval, or by using interval = x where x is the time interval in minutes used in the WeatherCat monthly .cat file(s). The most appropriate setting will depend on the completeness and (time) accuracy of the WeatherCat data being imported.

The default is derive.

qc

Determines whether simple quality control checks are applied to imported data. This option is identical in operation to the CSV qc option but applies to WeatherCat imports only. The default is True.

calc_missing

decimal

The character used as the decimal point in the WeatherCat monthly .cat files. This parameter must be included in quotation marks. Default is '.'.

tranche

The number of records written to the WeeWX database in each transaction. This option is identical in operation to the CSV tranche option but applies to WeatherCat imports only. The default is 250 which should suit most users.

UV_sensor

solar_sensor

[[Units]]

The [[Units]] stanza defines the units used in the WeatherCat monthly .cat files. Unit settings are required for temperature, pressure, rain and speed. The format for each setting is:

obs_type = weewx_unit_name

Where obs_type is one of temperature, pressure, rain or speed and weewx_unit_name is the WeeWX unit name of the units used by that particular obs_type (refer to the Units appendix in the Customization Guide for details of available WeeWX unit names). As WeatherCat supports a different suite of possible units only a subset of the available WeeWX unit names can be used for some settings.

Importing from CSV files

Warning!
Running WeeWX during a wee_import session can lead to abnormal termination of the import. If WeeWX must remain running (e.g., so that live data is not lost) run the wee_import session on another machine or to a second database and merge the in-use and second database once the import is complete.

wee_import can import data from a single CSV file. The CSV source file must be structured as follows:

The file must have a header row consisting of a comma separated list of field names. The field names can be any valid string as long as each field name is unique within the list. There is no requirement for the field names to be in any particular order as long as the same order is used for the observations on each row in the file. These field names will be mapped to WeeWX field names in the [CSV] section of the import configuration file.
Observation data for a given date-time must be listed on a single line with individual fields separated by a comma. The fields must be in the same order as the field names in the header row.
Blank fields are represented by the use of white space or no space only between commas.
Direction data being imported may be represented as numeric degrees or as a string representing the cardinal, intercardinal and/or secondary intercardinal directions.
There must a field that represents the date-time of the observations on each line. This date-time field must be either a Unix epoch timestamp or any date-time format that can be represented using Python strptime() format codes.

A CSV file suitable for import by wee_import may look like this:

Time,Barometer,Temp,Humidity,Windspeed,Dir,Gust,Dayrain,Radiation,Uv
28/11/2017 08:00:00,1016.9,24.6,84,1.8,113,8,0,359,3.8
28/11/2017 08:05:00,1016.9,25.1,82,4.8,135,11.3,0,775,4.7
28/11/2017 08:10:00,1016.9,25.4,80,4.4,127,11.3,0,787,5.1
28/11/2017 08:15:00,1017,25.7,79,3.5,74,11.3,0,800,5.4
28/11/2017 08:20:00,1016.9,25.9,79,1.6,95,9.7,0,774,5.5
28/11/2017 08:25:00,1017,25.5,78,2.9,48,9.7,0,303,3.4
28/11/2017 08:30:00,1017.1,25.1,80,3.1,54,9.7,0,190,3.6

or this:

Time,Barometer,Temp,Humidity,Windspeed,Dir,Gust,Dayrain,Radiation,Uv
2/1/2017 06:20:00,1006.4,4.8,48,2.8,NE,4,0,349,2.8
2/1/2017 06:25:00,1006.9,5.0,48,3.8,NNE,21.3,0,885,4.3
2/1/2017 06:30:00,1006.8,5.4,47,3.4,North,12.3,0,887,5.3
2/1/2017 06:35:00,1007,5.2,49,5.5,NNE,13.3,0,600,5.4
2/1/2017 06:40:00,1006.9,5.7,49,2.6,ESE,9.7,0,732,5.5
2/1/2017 06:45:00,1007,5.5,48,1.9,Southsoutheast,9.8,0,393,6.4
2/1/2017 06:50:00,1007.1,5.2,50,2.1,southeast,9.9,0,180,6.6

Note
Cardinal, intercardinal and/or secondary intercardinal directions may be represented by one, two or three letter abbreviations e.g., N, SE or SSW; by a single word e.g., North, Southwest or Southsouthwest or by hyphenated or spaced words e.g., North West or South-south-west. Capitalisation is ignored as are any spaces, hyphens or other white space. At present only English abbreviations and directions are supported.

Mapping data to archive fields

The WeeWX archive fields populated during a CSV import depend on the CSV-to-WeeWX field mappings specified in [[FieldMap]] stanza in the import configuration file. If a valid field mapping exists, the WeeWX field exists in the WeeWX archive table schema and provided the mapped CSV field contains valid data, then the corresponding WeeWX field will populated. Note that the CSV import is the only import supported by wee_import that allows any WeeWX archive field to be populated.

Note
The use of the calc_missing option in the import configuration file may result in a number of derived fields being calculated from the imported data. If these derived fields exist in the in-use database schema they will be saved to the database as well.

Step-by-step instructions

To import observations from a CSV file:

Ensure the source data file is in a directory accessible by the machine that will run wee_import. For the purposes of the following examples the source data file data.csv located in the /var/tmp directory will be used.
Make a backup of the WeeWX database in case the import should go awry.
Create an import configuration file. In this case we will make a copy of the example CSV import configuration file and save it as csv.conf in the /var/tmp directory:
```
$ cp /home/weewx/util/import/csv-example.conf /var/tmp/csv.conf
```
Confirm that the source option is set to CSV:
```
source = CSV
```
Confirm that the following options in the [CSV] section are set:
- file. The full path and file name of the file containing the CSV formatted data to be imported.
- delimiter. The single character used to separate fields.
- interval. Determines how the WeeWX interval field is derived.
- qc. Determines whether quality control checks are performed on the imported data.
- calc_missing. Determines whether missing derived observations will be calculated from the imported data.
- ignore_invalid_data. Determines whether invalid data in a source field is ignored or the import aborted.
- tranche. The number of records written to the WeeWX database in each transaction.
- UV_sensor. Whether a UV sensor was installed when the source data was produced.
- solar_sensor. Whether a solar radiation sensor was installed when the source data was produced.
- raw_datetime_format. The format of the imported date time field.
- rain. Determines how the WeeWX rain field is derived.
- wind_direction. Determines how imported wind direction fields are interpreted.
- [[FieldMap]]. Defines the mapping between imported data fields and WeeWX archive fields. Also defines the units of measure for each imported field.
When first importing data it is prudent to do a dry run import before any data are actually imported. A dry run import will perform all steps of the import without actually writing imported data to the WeeWX database. In addition, consideration should be given to any additional options such as --date.
To perform a dry run enter the following command:
```
wee_import --import-config=/var/tmp/csv.conf --dry-run
```
The output should be something like this:
```
Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/var/tmp/data.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
27337 records identified for import.
Unique records processed: 27337; Last timestamp: 2018-03-03 06:00:00 AEST (1520020800)
Finished dry run import
27337 records were processed and 27337 unique records would have been imported.
```
The output includes details about the data source, the destination of the imported data and some other details on how the data will be processed. The import will then be performed but no data will be written to the WeeWX database. Upon completion a brief summary of the records processed is provided.

Once the dry run results are satisfactory the data can be imported using the following command:

wee_import --import-config=/var/tmp/csv.conf

This will result in a short preamble similar to that from the dry run. At the end of the preamble there will be a prompt:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/var/tmp/data.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
27337 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?

If the import parameters are acceptable enter y to proceed with the import or n to abort the import. If the import is confirmed then the source data will be imported, processed and saved in the WeeWX database. Information on the progress of the import will be displayed similar to the following:
```
Unique records processed: 3250; Last timestamp: 2017-12-09 14:45:00 AEST (1512794700)
```
The line commencing with Unique records processed should update as records are imported with progress information on number of records processed, number of unique records imported and the date time of the latest record processed. Once the initial import is complete wee_import will, if requested, calculate any missing derived observations and rebuild the daily summaries. A brief summary should be displayed similar to the following:
```
Calculating missing derived observations...
Processing record: 27337; Last record: 2018-03-03 06:00:00 AEST (1520020800)
Recalculating daily summaries...
Records processed: 27337; Last date: 2018-03-03 06:00:00 AEST (1520020800)
Finished recalculating daily summaries
Finished calculating missing derived observations
```
When the import is complete a brief summary is displayed similar to the following:
```
Finished import
27337 records were processed and 27337 unique records imported in 113.91 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
```
Whilst wee_import will advise of the number of records processed and the number of unique records found, wee_import does know how many, if any, of the imported records were successfully saved to the database. You should look carefully through the WeeWX log file covering the wee_import session and take note of any records that were not imported. The most common reason for imported records not being saved to the database is because a record with that timestamp already exists in the database, in such cases something similar to the following will be found in the log:
```
Aug 22 14:38:28 stretch12  wee_import[1226] ERROR weewx.manager: Unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
```
In such cases you should take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.

Importing from Weather Underground

wee_import can import historical observation data for a Weather Underground PWS via the Weather Underground API. The Weather Underground API provides historical weather station observations received by Weather Underground for the PWS concerned on a day by day basis. As such, the data is analogous to the WeeWX archive table. When wee_import imports data from the Weather Underground API each day is considered a 'period'. wee_import processes one period at a time in chronological order (oldest to newest) and provides import summary data on a per period basis.

Mapping data to archive fields

A Weather Underground import will populate WeeWX archive fields as follows:

Provided data exists for each field returned by the Weather Underground API, the following WeeWX archive fields will be directly populated by imported data:
- dateTime
- barometer
- dewpoint
- heatindex
- outHumidity
- outTemp
- radiation
- rain
- rainRate
- UV
- windchill
- windDir
- windGust
- windSpeed
Note
If an appropriate field is not returned by the Weather Underground API then the corresponding WeeWX archive field will contain no data. If the API returns an appropriate field but with no data, the corresponding WeeWX archive field will be set to None/null. For example, if the API response has no solar radiation field the WeeWX radiation archive field will have no data stored. However, if the API response has a solar radiation field but contains no data, the WeeWX radiation archive field will be None/null.
The following WeeWX archive fields will be populated from other settings or configuration options:
- interval
- usUnits
The following WeeWX archive fields will be populated with values derived from the imported data provided calc_missing = True is included in the [WU] section of the import configuration file and the field exists in the in-use WeeWX archive table schema.
- altimeter
- ET
- pressure
Note
If calc_missing = False is included in the [WU] section of the import configuration file being used then all of the above fields will be set to None/null. The default setting of the calc_missing option is True

Step-by-step instructions

To import observations from a Weather Underground PWS history:

Obtain the weather station ID of the Weather Underground PWS from which data is to be imported. The station ID will be a sequence of numbers and upper case letters that is usually 11 or 12 characters in length. For the purposes of the following examples a weather station ID of ISTATION123 will be used.
Obtain the API key to be used to access the Weather Underground API. This will be a seemingly random alphanumeric sequence of 32 characters. API keys are available to Weather Underground PWS contributors by logging on to their Weather Underground account and accessing Member Settings.
Make a backup of the WeeWX database in case the import should go awry.
Create an import configuration file. In this case we will make a copy of the example Weather Underground import configuration file and save it as wu.conf in the /var/tmp directory:
```
$ cp /home/weewx/util/import/wu-example.conf /var/tmp/wu.conf
```
Confirm that the source option is set to WU
```
source = WU
```
Confirm that the following options in the [WU] section are correctly set:
- station_id. The 11 or 12 character weather station ID of the Weather Underground PWS that will be the source of the imported data.
- api_key. The 32 character API key to be used to access the Weather Underground API.
- interval. Determines how the WeeWX interval field is derived.
- qc. Determines whether quality control checks are performed on the imported data.
  Note
  As Weather Underground imports at times contain nonsense values, particularly for fields for which no data were uploaded to Weather Underground by the PWS, the use of quality control checks on imported data can prevent these nonsense values from being imported and contaminating the WeeWX database.
- calc_missing. Determines whether missing derived observations will be calculated from the imported data.
- ignore_invalid_data. Determines whether invalid data in a source field is ignored or the import aborted.
- tranche. The number of records written to the WeeWX database in each transaction.
- wind_direction. Determines how imported wind direction fields are interpreted.
When first importing data it is prudent to do a dry run import before any data is actually imported. A dry run import will perform all steps of the import without actually writing imported data to the WeeWX database. In addition, consideration should be given to any additional options to be used such as --date, --from or --to.
To perform a dry run enter the following command:
```
wee_import --import-config=/var/tmp/wu.conf --from=2016-01-20T22:30 --to=2016-01-23T06:00 --dry-run
```
In this case the --from and --to options have been used to import Weather Underground records from 10:30pm on 20 January 2016 to 6:00am on 23 January 2016 inclusive.

Note
If the --date option is omitted, or a date (not date-time) range is specified using the --from and --to options during a Weather Underground import, then one or more full days of history data will be imported. This includes records timestamped from 00:00 (inclusive) at the start of the day up to but NOT including the 00:00 record at the end of the last day. As the timestamped record refers to observations of the previous interval, such an import actually includes one record with observations from the previous day (the 00:00 record at the start of the day). Whilst this will not present a problem for wee_import as any records being imported with a timestamp that already exists in the WeeWX database are ignored, you may wish to use the --from and --to options with a suitable date-time range to precisely control which records are imported.

Note
wee_import obtains Weather Underground daily history data one day at a time via a HTTP request and as such the import of large time spans of data may take some time. Such imports may be best handled as a series of imports of smaller time spans.

This will result in a short preamble with details on the data source, the destination of the imported data and some other details on how the data will be processed. The import will then be performed but no data will written to the WeeWX database.

The output should be similar to:
```
Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Observation history for Weather Underground station 'ISTATION123' will be imported.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Observations timestamped after 2016-01-20 22:30:00 AEST (1453293000) and up to and
including 2016-01-23 06:00:00 AEST (1453492800) will be imported.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Unique records processed: 18; Last timestamp: 2016-01-20 23:55:00 AEST (1453298100)
Period 2 ...
Unique records processed: 284; Last timestamp: 2016-01-21 23:55:00 AEST (1453384500)
Period 3 ...
Unique records processed: 284; Last timestamp: 2016-01-22 23:55:00 AEST (1453470900)
Period 4 ...
Unique records processed: 71; Last timestamp: 2016-01-23 06:00:00 AEST (1453492800)
Finished dry run import
657 records were processed and 657 unique records would have been imported.
```
Note
Any periods for which no data could be obtained will be skipped. The lack of data may be due to an incorrect station ID, an incorrect date or Weather Underground API problems. A short explanatory note to this effect will be displayed against the period concerned and an entry included in the log.

Once the dry run results are satisfactory the source data can be imported using the following command:

wee_import --import-config=/var/tmp/wu.conf --from=2016-01-20T22:30 --to=2016-01-23T06:00

This will result in a short preamble similar to that of a dry run. At the end of the preamble there will be a prompt:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Observation history for Weather Underground station 'ISTATION123' will be imported.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Observations timestamped after 2016-01-20 22:30:00 AEST (1453293000) and up to and
including 2016-01-23 06:00:00 AEST (1453492800) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?

Note
wee_import obtains Weather Underground data one day at a time via a HTTP request and as such the import of large time spans of data may take some time. Such imports may be best handled as a series of imports of smaller time spans.

If the import parameters are acceptable enter y to proceed with the import or n to abort the import. If the import is confirmed, the source data will be imported, processed and saved in the WeeWX database. Information on the progress of the import will be displayed similar to the following:
```
Unique records processed: 18; Last timestamp: 2016-01-20 23:55:00 AEST (1453298100)
Period 2 ...
Unique records processed: 284; Last timestamp: 2016-01-21 23:55:00 AEST (1453384500)
Period 3 ...
Unique records processed: 284; Last timestamp: 2016-01-22 23:55:00 AEST (1453470900)
```
Note
Any periods for which no data could be obtained will be skipped. The lack of data may be due to an incorrect station ID, an incorrect date or Weather Underground API problems. A short explanatory note to this effect will be displayed against the period concerned and an entry included in the log.

The line commencing with Unique records processed should update as records are imported with progress information on number of records processed, number of unique records imported and the date time of the latest record processed. If the import spans multiple days then a new Period line is created for each day.

Once the initial import is complete wee_import will, if requested, calculate any missing derived observations and rebuild the daily summaries. A brief summary should be displayed similar to the following:
```
Calculating missing derived observations ...
Processing record: 204; Last record: 2016-01-22 23:55:00 AEST (1453470900)
Recalculating daily summaries...
Finished recalculating daily summaries
Finished calculating missing derived observations
```
When the import is complete a brief summary is displayed similar to the following:
```
Finished import
657 records were processed and 657 unique records imported in 78.97 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
```
Note
The new (2019) Weather Underground API appears to have an issue when obtaining historical data for the current day. The first time the API is queried the API returns all historical data up to and including the most recent record. However, subsequent later API queries during the same day return the same set of records rather than all records up to and including the time of the latest API query. Users importing Weather Underground data that includes data from the current day are advised to carefully check the WeeWX log to ensure that all expected records were imported. If some records are missing from the current day try running an import for the current day again using the --date option setting. If this fails then wait until the following day and perform another import for the day concerned again using the --date option setting. In all cases confirm what data has been imported by referring to the WeeWX log.
Whilst wee_import will advise of the number of records processed and the number of unique records found, wee_import does know how many, if any, of the imported records were successfully saved to the database. You should look carefully through the WeeWX log file covering the wee_import session and take note of any records that were not imported. The most common reason for imported records not being saved to the database is because a record with that timestamp already exists in the database, in such cases something similar to the following will be found in the log:
```
Aug 22 14:38:28 stretch12 weewx[863]: manager: unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
```
In such cases you should take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.

Importing from Cumulus

wee_import can import observational data from the one or more Cumulus monthly log files. A Cumulus monthly log file records weather station observations for a single month. These files are accumulated over time and can be considered analogous to the WeeWX archive table. When wee_import imports data from the Cumulus monthly log files each log file is considered a 'period'. wee_import processes one period at a time in chronological order (oldest to newest) and provides import summary data on a per period basis.

Mapping data to archive fields

A Cumulus monthly log file import will populate the WeeWX archive fields as follows:

Provided data exists for each field in the Cumulus monthly logs, the following WeeWX archive fields will be directly populated by imported data:
- dateTime
- barometer
- dewpoint
- heatindex
- inHumidity
- inTemp
- outHumidity
- outTemp
- radiation
- rain
- rainRate
- UV
- windDir
- windGust
- windSpeed
- windchill
Note
If a field in the Cumulus monthly log file has no data then the corresponding WeeWX archive field will be set to None/null.
The following WeeWX archive fields will be populated from other settings or configuration options:
- interval
- usUnits
The following WeeWX archive fields will be populated with values derived from the imported data provided calc_missing = True is included in the [Cumulus] section of the import configuration file being used and the field exists in the in-use WeeWX archive table schema.
- altimeter
- ET
- pressure
Note
If calc_missing = False is included in the [Cumulus] section of the import configuration file being used then all of the above fields will be set to None/null. The default setting of the calc_missing option is True

Step-by-step instructions

To import observations from one or more Cumulus monthly log files:

Ensure the Cumulus monthly log file(s) to be used for the import are located in a directory accessible by the machine that will run wee_import. For the purposes of the following examples, there are nine monthly logs files covering the period October 2016 to June 2017, inclusive, located in the /var/tmp/cumulus directory.
Make a backup of the WeeWX database in case the import should go awry.
Create an import configuration file. In this case we will make a copy of the example Cumulus import configuration file and save it as cumulus.conf in the /var/tmp directory:
```
$ cp /home/weewx/util/import/cumulus-example.conf /var/tmp/cumulus.conf
```
Confirm that the source option is set to Cumulus:
```
source = Cumulus
```
Confirm that the following options in the [Cumulus] section are correctly set:
- directory. The full path to the directory containing the Cumulus monthly log files to be used as the source of the imported data.
- interval. Determines how the WeeWX interval field is derived.
- qc. Determines whether quality control checks are performed on the imported data.
- calc_missing. Determines whether missing derived observations will be calculated from the imported data.
- separator. The date field separator used in the Cumulus monthly log files.
- delimiter. The field delimiter used in the Cumulus monthly log files.
- decimal. The decimal point character used in the Cumulus monthly log files.
- ignore_invalid_data. Determines whether invalid data in a source field is ignored or the import aborted.
- tranche. The number of records written to the WeeWX database in each transaction.
- UV_sensor. Whether a UV sensor was installed when the source data was produced.
- solar_sensor. Whether a solar radiation sensor was installed when the source data was produced.
- [[Units]]. Defines the units used in the Cumulus monthly log files.

When first importing data it is prudent to do a dry run import before any data is actually imported. A dry run import will perform all steps of the import without actually writing imported data to the WeeWX database. In addition, consideration should be given to any additional options to be used such as --date.

To perform a dry run enter the following command:

wee_import --import-config=/var/tmp/cumulus.conf --dry-run

This will result in a short preamble with details on the data source, the destination of the imported data and some other details on how the data will be processed. The import will then be performed but no data will be written to the WeeWX database.

The output should be similar to:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Unique records processed: 8858; Last timestamp: 2016-10-31 23:55:00 AEST (1477922100)
Period 2 ...
Unique records processed: 8636; Last timestamp: 2016-11-30 23:55:00 AEST (1480514100)
Period 3 ...
Unique records processed: 8925; Last timestamp: 2016-12-31 23:55:00 AEST (1483192500)
Period 4 ...
Unique records processed: 8908; Last timestamp: 2017-01-31 23:55:00 AEST (1485870900)
Period 5 ...
Unique records processed: 8029; Last timestamp: 2017-02-28 23:55:00 AEST (1488290100)
Period 6 ...
Unique records processed: 8744; Last timestamp: 2017-03-31 23:55:00 AEST (1490968500)
Period 7 ...
Unique records processed: 8489; Last timestamp: 2017-04-30 23:02:00 AEST (1493557320)
Period 8 ...
Unique records processed: 8754; Last timestamp: 2017-05-31 23:55:00 AEST (1496238900)
Period 9 ...
Unique records processed: 8470; Last timestamp: 2017-06-30 23:55:00 AEST (1498830900)
Finished dry run import
77813 records were processed and 77813 unique records would have been imported.

Note
The nine periods correspond to the nine monthly log files used for this import.

Note
Any periods for which no data could be obtained will be skipped. The lack of data may be due to a missing Cumulus monthly log file. A short explanatory note to this effect will be displayed against the period concerned and an entry included in the log.

Once the dry run results are satisfactory the data can be imported using the following command:

wee_import --import-config=/var/tmp/cumulus.conf

This will result in a preamble similar to that of a dry run. At the end of the preamble there will be a prompt:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?

If there is more than one Cumulus monthly log file then wee_import will provide summary information on a per period basis during the import. In addition, if the --date option is used then source data that falls outside the date or date range specified with the --date option is ignored. In such cases the preamble may look similar to:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Period 1 - no records identified for import.
Period 2 ...
Period 2 - no records identified for import.
Period 3 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?

If the import parameters are acceptable enter y to proceed with the import or n to abort the import. If the import is confirmed, the source data will be imported, processed and saved in the WeeWX database. Information on the progress of the import will be displayed similar to the following:
```
Unique records processed: 2305; Last timestamp: 2016-12-30 00:00:00 AEST (1483020000)
```
Again if there is more than one Cumulus monthly log file and if the --date option is used then the progress information may instead look similar to:
```
Period 4 ...
Unique records processed: 8908; Last timestamp: 2017-01-31 23:55:00 AEST (1485870900)
Period 5 ...
Unique records processed: 8029; Last timestamp: 2017-02-28 23:55:00 AEST (1488290100)
Period 6 ...
Unique records processed: 8744; Last timestamp: 2017-03-31 23:55:00 AEST (1490968500)
```
Note
Any periods for which no data could be obtained will be skipped. The lack of data may be due to a missing Cumulus monthly log file. A short explanatory note to this effect will be displayed against the period concerned and an entry included in the log.

The line commencing with Unique records processed should update as records are imported with progress information on number of records processed, number of unique records imported and the date time of the latest record processed. If the import spans multiple months (ie multiple monthly log files) then a new Period line is created for each month.

Once the initial import is complete wee_import will, if requested, calculate any missing derived observations and rebuild the daily summaries. A brief summary should be displayed similar to the following:
```
Calculating missing derived observations ...
Processing record: 77782; Last record: 2017-06-30 00:00:00 AEST (1519826400)
Recalculating daily summaries...
Records processed: 77000; Last date: 2017-06-28 11:45:00 AEST (1519811100)
Finished recalculating daily summaries
Finished calculating missing derived observations
```
When the import is complete a brief summary is displayed similar to the following:
```
Finished import
77813 records were processed and 77813 unique records imported in 106.96 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
```
Whilst wee_import will advise of the number of records processed and the number of unique records found, wee_import does know how many, if any, of the imported records were successfully saved to the database. You should look carefully through the WeeWX log file covering the wee_import session and take note of any records that were not imported. The most common reason for imported records not being saved to the database is because a record with that timestamp already exists in the database, in such cases something similar to the following will be found in the log:
```
Aug 22 14:38:28 stretch12 weewx[863]: manager: unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
```
In such cases take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.

Importing from Weather Display

wee_import can import observational data from the one or more Weather Display monthly log files. Weather Display records observational data on a monthly basis in a number of either space delimited (.txt) and/or comma separated (.csv) text files. wee_import can import observational data from the following Weather Display log files:

MMYYYYlg.txt
MMYYYYlgcsv.csv (csv format version of MMYYYYlg.txt)
MMYYYYvantagelog.txt
MMYYYYvantagelogcsv.csv (csv format version of MMYYYYvantagelog.txt)
MMYYYYvantageextrasensorslog.csv

where MM is a one or two digit month and YYYY is a four digit year

The Weather Display monthly log files record observational data using a nominal 1 minute interval with each file recording various observations for the month and year designated by the MM and YYYY components of the file name. These files are accumulated over time and can be considered analogous to the WeeWX archive table. When wee_import imports data from the Weather Display monthly log files each set of log files for a given month and year is considered a 'period'. wee_import processes one period at a time in chronological order (oldest to newest) and provides import summary data on a per period basis.

Mapping data to archive fields

The WeeWX archive fields populated during the import of Weather Display data depends on the field mapping specified in [[FieldMap]] stanza in the import configuration file. A given WeeWX field will be populated if:

a valid field mapping exists,
the WeeWX field exists in the WeeWX archive table schema, and
the mapped Weather Display field contains valid data.

The following WeeWX archive fields will be populated from other settings or configuration options and need not be included in the field map:

interval
usUnits

The following WeeWX archive fields will be populated with values derived from the imported data provided calc_missing = True is included in the [WD] section of the import configuration file being used and the field exists in the in-use WeeWX archive table schema:

altimeter
pressure
rainRate
windchill

Note
If calc_missing = False is included in the [WD] section of the import configuration file being used then all of the above fields will be set to None/null. The default setting of the calc_missing option is True

Step-by-step instructions

To import observations from one or more Weather Display monthly log files:

Ensure the Weather Display monthly log file(s) to be used for the import are located in a directory accessible by the machine that will run wee_import. For the purposes of the following examples, there are five months of logs files covering the period September 2018 to January 2019 inclusive located in the /var/tmp/wd directory.
Make a backup of the WeeWX database in case the import should go awry.
Create an import configuration file, this is easily done by making a copy of the example Weather Display import configuration file located in the /home/weewx/util/import directory. In this case we will make a copy of the example Weather Display import configuration file and save it as wd.conf in the /var/tmp directory:
```
$ cp /home/weewx/util/import/wd-example.conf /var/tmp/wd.conf
```
Confirm that the source option is set to WD:
```
source = WD
```
Confirm that the following options in the [WD] section are correctly set:
- directory. The full path to the directory containing the Weather Display monthly log files to be used as the source of the imported data.
- logs_to_process. Specifies the Weather Display monthly log files to be used to import data.
- interval. Determines how the WeeWX interval field is derived.
- qc. Determines whether quality control checks are performed on the imported data.
- calc_missing. Determines whether missing derived observations will be calculated from the imported data.
- txt_delimiter. The field delimiter used in the Weather Display space delimited (*.txt) monthly log files.
- csv_delimiter. The field delimiter used in the Weather Display monthly comma separated values (*.csv) monthly log files.
- decimal. The decimal point character used in the Weather Display monthly log files.
- ignore_missing_log. Determines whether missing log files are to be ignored or the import aborted.
- ignore_invalid_data. Determines whether invalid data in a source field is ignored or the import aborted.
- tranche. The number of records written to the WeeWX database in each transaction.
- UV_sensor. Whether a UV sensor was installed when the source data was produced.
- solar_sensor. Whether a solar radiation sensor was installed when the source data was produced.
- ignore_extreme_temp_hum. Determines whether temperature and humidity values of 255 will be ignored.
- [[Units]]. Defines the units used in the Weather Display monthly log files.
- [[FieldMap]]. Defines the mapping between imported data fields and WeeWX archive fields.
When first importing data it is prudent to do a dry run import before any data is actually imported. A dry run import will perform all steps of the import without actually writing imported data to the WeeWX database. In addition, consideration should be given to any additional options to be used such as --date.
Note
Due to some peculiarities of the Weather Display log structure it may be prudent to use the --suppress--warnings option during the initial dry run so the overall progress of the import can be observed.

To perform a dry run enter the following command:
```
wee_import --import-config=/var/tmp/wd.conf --dry-run --suppress-warnings
```
This will result in a short preamble with details on the data source, the destination of the imported data and some other details on how the data will be processed. The import will then be performed but no data will be written to the WeeWX database.

The output should be similar to:
```
Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Weather Display monthly log files in the '/var/tmp/WD' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Unique records processed: 43183; Last timestamp: 2018-09-30 23:59:00 AEST (1538315940)
Period 2 ...
Unique records processed: 44620; Last timestamp: 2018-10-31 23:59:00 AEST (1540994340)
Period 3 ...
Unique records processed: 43136; Last timestamp: 2018-11-30 23:59:00 AEST (1543586340)
Period 4 ...
Unique records processed: 44633; Last timestamp: 2018-12-31 23:59:00 AEST (1546264740)
Period 5 ...
Unique records processed: 8977; Last timestamp: 2019-01-07 05:43:00 AEST (1546803780)
Finished dry run import
184765 records were processed and 184549 unique records would have been imported.
216 duplicate records were ignored.
```
Note
The five periods correspond to the five months of log files used for this import.

Note
Any periods for which no data could be obtained will be skipped. The lack of data may be due to a missing Weather Display log file. A short explanatory note to this effect will be displayed against the period concerned and an entry included in the log.

If the --suppress--warnings option was used it may be prudent to do a second dry run this time without the --suppress--warnings option. This will allow any warnings generated by the dry run import to be observed:

wee_import --import-config=/var/tmp/wd.conf --dry-run

The output should be similar to:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Weather Display monthly log files in the '/var/tmp/WD' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Warning: Import field 'radiation' is mapped to WeeWX field 'radiation' but the
         import field 'radiation' could not be found in one or more records.
         WeeWX field 'radiation' will be set to 'None' in these records.
Warning: Import field 'soiltemp' is mapped to WeeWX field 'soilTemp1' but the
         import field 'soiltemp' could not be found in one or more records.
         WeeWX field 'soilTemp1' will be set to 'None' in these records.
Warning: Import field 'soilmoist' is mapped to WeeWX field 'soilMoist1' but the
         import field 'soilmoist' could not be found in one or more records.
         WeeWX field 'soilMoist1' will be set to 'None' in these records.
Warning: Import field 'humidity' is mapped to WeeWX field 'outHumidity' but the
         import field 'humidity' could not be found in one or more records.
         WeeWX field 'outHumidity' will be set to 'None' in these records.
Warning: Import field 'heatindex' is mapped to WeeWX field 'heatindex' but the
         import field 'heatindex' could not be found in one or more records.
         WeeWX field 'heatindex' will be set to 'None' in these records.
Warning: Import field 'windspeed' is mapped to WeeWX field 'windSpeed' but the
         import field 'windspeed' could not be found in one or more records.
         WeeWX field 'windSpeed' will be set to 'None' in these records.
Warning: Import field 'barometer' is mapped to WeeWX field 'barometer' but the
         import field 'barometer' could not be found in one or more records.
         WeeWX field 'barometer' will be set to 'None' in these records.
Warning: Import field 'dewpoint' is mapped to WeeWX field 'dewpoint' but the
         import field 'dewpoint' could not be found in one or more records.
         WeeWX field 'dewpoint' will be set to 'None' in these records.
Warning: Import field 'rainlastmin' is mapped to WeeWX field 'rain' but the
         import field 'rainlastmin' could not be found in one or more records.
         WeeWX field 'rain' will be set to 'None' in these records.
Warning: Import field 'direction' is mapped to WeeWX field 'windDir' but the
         import field 'direction' could not be found in one or more records.
         WeeWX field 'windDir' will be set to 'None' in these records.
Warning: Import field 'temperature' is mapped to WeeWX field 'outTemp' but the
         import field 'temperature' could not be found in one or more records.
         WeeWX field 'outTemp' will be set to 'None' in these records.
Warning: Import field 'gustspeed' is mapped to WeeWX field 'windGust' but the
         import field 'gustspeed' could not be found in one or more records.
         WeeWX field 'windGust' will be set to 'None' in these records.
Unique records processed: 43183; Last timestamp: 2018-09-30 23:59:00 AEST (1538315940)
Period 2 ...
Unique records processed: 44620; Last timestamp: 2018-10-31 23:59:00 AEST (1540994340)
Period 3 ...
Unique records processed: 43136; Last timestamp: 2018-11-30 23:59:00 AEST (1543586340)
Period 4 ...
Unique records processed: 44633; Last timestamp: 2018-12-31 23:59:00 AEST (1546264740)
Period 5 ...
Unique records processed: 8977; Last timestamp: 2019-01-07 05:43:00 AEST (1546803780)
    6 duplicate records were identified in period 5:
        2019-01-04 10:31:00 AEST (1546561860)
        2019-01-04 10:32:00 AEST (1546561920)
        2019-01-04 10:33:00 AEST (1546561980)
        2019-01-04 10:34:00 AEST (1546562040)
        2019-01-04 10:35:00 AEST (1546562100)
        2019-01-04 10:36:00 AEST (1546562160)
Finished dry run import
184555 records were processed and 184549 unique records would have been imported.
6 duplicate records were ignored.

In this case the following warnings are evident:

Period one had 12 warnings for import fields that were mapped to WeeWX data fields but for which no data was found. This could be a sign that a complete month of data or a significant portion of the month could be missing or it could be a case of just the first record of the month is missing (a significant number of Weather Display monthly log files have been found to be missing the first record of the month). In most cases this warning can be ignored.
Period five shows warnings for six entries in the period that have duplicate timestamps. This could be a sign that there is a problem in one or more of the Weather Display monthly log files for that month. However, anecdotally it has been found that duplicate entries often exist in one or more Weather Display monthly log files. If the duplicates are to be ignored then such warnings can be ignored otherwise the incorrect data should be removed from the affected log files before import.

Once the dry run results are satisfactory the data can be imported using the following command:

wee_import --import-config=/var/tmp/wd.conf --suppress-warnings

Note
The --suppress--warnings option has been used to suppress the previously encountered warnings.

This will result in a preamble similar to that of a dry run. At the end of the preamble there will be a prompt:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Weather Display monthly log files in the '/var/tmp/WD' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?

If there is more than one month of Weather Display monthly log files then wee_import will provide summary information on a per period basis during the import. In addition, if the --date option is used then source data that falls outside the date or date range specified with the --date option is ignored. In such cases the preamble may look similar to:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Weather Display monthly log files in the '/var/tmp/WD' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Observations timestamped after 2018-10-12 00:00:00 AEST (1539266400) and up to and
including 2018-10-13 00:00:00 AEST (1539352800) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Period 1 - no records identified for import.
Period 2 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?

If the import parameters are acceptable enter y to proceed with the import or n to abort the import. If the import is confirmed, the source data will be imported, processed and saved in the WeeWX database. Information on the progress of the import will be displayed similar to the following:
```
Unique records processed: 1250; Last timestamp: 2018-12-01 20:49:00 AEST (1543661340)
```
Again if there is more than one month of Weather Display monthly log files and if the --date option is used then the progress information may instead look similar to:
```
Period 2 ...
Unique records processed: 44620; Last timestamp: 2018-10-31 23:59:00 AEST (1540994340)
Period 3 ...
Unique records processed: 43136; Last timestamp: 2018-11-30 23:59:00 AEST (1543586340)
Period 4 ...
Unique records processed: 12000; Last timestamp: 2018-12-09 07:59:00 AEST (1544306340)
```
Note
Any periods for which no data could be obtained will be skipped. The lack of data may be due to a missing Weather Display log file. A short explanatory note to this effect will be displayed against the period concerned and an entry included in the log.

The line commencing with Unique records processed should update as records are imported with progress information on number of unique records processed and the date time of the latest record processed. If the import spans multiple months then a new Period line is created for each month.

Once the initial import is complete wee_import will, if requested, calculate any missing derived observations and rebuild the daily summaries. A brief summary should be displayed similar to the following:
```
Calculating missing derived observations ...
Processing record: 184549; Last record: 2019-01-08 00:00:00 AEST (1546869600)
Recalculating daily summaries...
Records processed: 184000; Last date: 2019-01-06 20:34:00 AEST (1546770840)
Finished recalculating daily summaries
Finished calculating missing derived observations
```
When the import is complete a brief summary is displayed similar to the following:
```
Finished import
184765 records were processed and 184549 unique records imported in 699.27 seconds.
216 duplicate records were ignored.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
```
Whilst wee_import will advise of the number of unique records imported, wee_import does know how many, if any, of the imported records were successfully saved to the database. You should look carefully through the WeeWX log file covering the wee_import session and take note of any records that were not imported. The most common reason for imported records not being saved to the database is because a record with that timestamp already exists in the database, in such cases something similar to the following will be found in the log:
```
Aug 22 14:38:28 stretch12 weewx[863]: manager: unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
```
In such cases take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.

Importing from WeatherCat

wee_import can import observational data from the one or more WeatherCat monthly .cat files. A WeatherCat monthly .cat file records weather station observations for a single month. These files are accumulated over time and can be considered analogous to the WeeWX archive table. When wee_import imports data from the WeatherCat monthly .cat files each file is considered a 'period'. wee_import processes one period at a time in chronological order (oldest to newest) and provides import summary data on a per period basis.

Mapping data to archive fields

A WeatherCat import will populate the WeeWX archive fields as follows:

Provided data exists for each field in the WeatherCat monthly .cat files, the following WeeWX archive fields will be directly populated by imported data:
- dateTime
- barometer
- dewpoint
- heatindex
- inHumidity
- inTemp
- outHumidity
- outTemp
- radiation
- rain
- rainRate
- UV
- windDir
- windGust
- windSpeed
- windchill
Note
If a field in the WeatherCat monthly .cat file has no data then the corresponding WeeWX archive field will be set to None/null.
The following WeeWX archive fields will be populated from other settings or configuration options:
- interval
- usUnits
The following WeeWX archive fields will be populated with values derived from the imported data provided calc_missing = True is included in the [WeatherCat] section of the import configuration file being used and the field exists in the in-use WeeWX archive table schema.
- altimeter
- ET
- pressure
Note
If calc_missing = False is included in the [WeatherCat] section of the import configuration file being used then all of the above fields will be set to None/null. The default setting of the calc_missing option is True

Step-by-step instructions

To import observations from one or more WeatherCat monthly .cat files:

Ensure the WeatherCat monthly .cat file(s) to be used for the import are located in year directories with the year directories in turn located in a directory accessible by the machine that will run wee_import. For the purposes of the following examples, there are nine monthly logs files covering the period October 2016 to June 2017 inclusive, located in the /var/tmp/wcat/2016 and /var/tmp/wcat/2017 directories respectively.
Make a backup of the WeeWX database in case the import should go awry.
Create an import configuration file. In this case we will make a copy of the example WeatherCat import configuration file and save it as wcat.conf in the /var/tmp directory:
```
$ cp /home/weewx/util/import/weathercat-example.conf /var/tmp/wcat.conf
```
Confirm that the source option is set to WeatherCat:
```
source = WeatherCat
```
Confirm that the following options in the [WeatherCat] section are correctly set:
- directory. The full path to the directory containing the directories containing the WeatherCat monthly .cat files to be used as the source of the imported data.
- interval. Determines how the WeeWX interval field is derived.
- qc. Determines whether quality control checks are performed on the imported data.
- calc_missing. Determines whether missing derived observations will be calculated from the imported data.
- decimal. The decimal point character used in the WeatherCat monthly log files.
- tranche. The number of records written to the WeeWX database in each transaction.
- UV_sensor. Whether a UV sensor was installed when the source data was produced.
- solar_sensor. Whether a solar radiation sensor was installed when the source data was produced.
- [[Units]]. Defines the units used in the WeatherCat monthly .cat files.
When first importing data it is prudent to do a dry run import before any data is actually imported. A dry run import will perform all steps of the import without actually writing imported data to the WeeWX database. In addition, consideration should be given to any additional options to be used such as --date.
Note
Whilst WeatherCat monthly .cat files use a fixed set of fields the inclusion of fields other than t (timestamp) and V (validation) is optional. For this reason the field map used for WeatherCat imports includes fields that may not exist in some WeatherCat monthly .cat files resulting in warnings by wee_port that there may be missing data in the import source. These warnings can be extensive and may detract from the ability of the user to monitor the progress of the import. It may be prudent to use the --suppress--warnings option during the initial dry run so the overall progress of the import can be more easily observed.

To perform a dry run enter the following command:
```
wee_import --import-config=/var/tmp/wcat.conf --dry-run --suppress-warnings
```
This will result in a short preamble with details on the data source, the destination of the imported data and some other details on how the data will be processed. The import will then be performed but no data will be written to the WeeWX database.

The output should be similar to:
```
Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
WeatherCat monthly .cat files in the '/var/tmp/wcat' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Unique records processed: 39555; Last timestamp: 2016-10-31 23:59:00 AEST (1477922340)
Period 2 ...
Unique records processed: 38284; Last timestamp: 2016-11-30 23:59:00 AEST (1480514340)
Period 3 ...
Unique records processed: 39555; Last timestamp: 2016-12-31 23:59:00 AEST (1483192740)
Period 4 ...
Unique records processed: 39555; Last timestamp: 2017-01-31 23:59:00 AEST (1485871140)
Period 5 ...
Unique records processed: 35598; Last timestamp: 2017-02-28 23:59:00 AEST (1488290340)
Period 6 ...
Unique records processed: 39555; Last timestamp: 2017-03-31 23:59:00 AEST (1490968740)
Period 7 ...
Unique records processed: 38284; Last timestamp: 2017-04-30 23:59:00 AEST (1493560740)
Period 8 ...
Unique records processed: 38284; Last timestamp: 2017-06-30 23:59:00 AEST (1498831140)
Finished dry run import
308670 records were processed and 308670 unique records would have been imported.
```
Note
The eight periods correspond to the eight monthly .cat files used for this import.

Note
Any periods for which no data could be obtained will be skipped. The lack of data may be due to a missing WeatherCat monthly .cat file. A short explanatory note to this effect will be displayed against the period concerned and an entry included in the log.

wee_import --import-config=/var/tmp/wcat.conf --dry-run

The output should be similar to:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
WeatherCat monthly .cat files in the '/var/tmp/wcat' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Warning: Import field 'T1' is mapped to WeeWX field 'extraTemp1' but the
         import field 'T1' could not be found in one or more records.
         WeeWX field 'extraTemp1' will be set to 'None' in these records.
Warning: Import field 'T2' is mapped to WeeWX field 'extraTemp2' but the
         import field 'T2' could not be found in one or more records.
         WeeWX field 'extraTemp2' will be set to 'None' in these records.
Warning: Import field 'T3' is mapped to WeeWX field 'extraTemp3' but the
         import field 'T3' could not be found in one or more records.
         WeeWX field 'extraTemp3' will be set to 'None' in these records.
Warning: Import field 'H1' is mapped to WeeWX field 'extraHumid1' but the
         import field 'H1' could not be found in one or more records.
         WeeWX field 'extraHumid1' will be set to 'None' in these records.
Warning: Import field 'H2' is mapped to WeeWX field 'extraHumid2' but the
         import field 'H2' could not be found in one or more records.
         WeeWX field 'extraHumid2' will be set to 'None' in these records.
Warning: Import field 'Sm1' is mapped to WeeWX field 'soilMoist1' but the
         import field 'Sm1' could not be found in one or more records.
         WeeWX field 'soilMoist1' will be set to 'None' in these records.
Warning: Import field 'Sm2' is mapped to WeeWX field 'soilMoist2' but the
         import field 'Sm2' could not be found in one or more records.
         WeeWX field 'soilMoist2' will be set to 'None' in these records.
Warning: Import field 'Sm3' is mapped to WeeWX field 'soilMoist3' but the
         import field 'Sm3' could not be found in one or more records.
         WeeWX field 'soilMoist3' will be set to 'None' in these records.
Warning: Import field 'Sm4' is mapped to WeeWX field 'soilMoist4' but the
         import field 'Sm4' could not be found in one or more records.
         WeeWX field 'soilMoist4' will be set to 'None' in these records.
Warning: Import field 'Lw1' is mapped to WeeWX field 'leafWet1' but the
         import field 'Lw1' could not be found in one or more records.
         WeeWX field 'leafWet1' will be set to 'None' in these records.
Warning: Import field 'Lw2' is mapped to WeeWX field 'leafWet2' but the
         import field 'Lw2' could not be found in one or more records.
         WeeWX field 'leafWet2' will be set to 'None' in these records.
Warning: Import field 'St1' is mapped to WeeWX field 'soilTemp1' but the
         import field 'St1' could not be found in one or more records.
         WeeWX field 'soilTemp1' will be set to 'None' in these records.
Warning: Import field 'St2' is mapped to WeeWX field 'soilTemp2' but the
         import field 'St2' could not be found in one or more records.
         WeeWX field 'soilTemp2' will be set to 'None' in these records.
Warning: Import field 'St3' is mapped to WeeWX field 'soilTemp3' but the
         import field 'St3' could not be found in one or more records.
         WeeWX field 'soilTemp3' will be set to 'None' in these records.
Warning: Import field 'St4' is mapped to WeeWX field 'soilTemp4' but the
         import field 'St4' could not be found in one or more records.
         WeeWX field 'soilTemp4' will be set to 'None' in these records.
Warning: Import field 'Lt1' is mapped to WeeWX field 'leafTemp1' but the
         import field 'Lt1' could not be found in one or more records.
         WeeWX field 'leafTemp1' will be set to 'None' in these records.
Warning: Import field 'Lt2' is mapped to WeeWX field 'leafTemp2' but the
         import field 'Lt2' could not be found in one or more records.
         WeeWX field 'leafTemp2' will be set to 'None' in these records.
Unique records processed: 39555; Last timestamp: 2016-10-31 23:59:00 AEST (1477922340)
Period 2 ...
Warning: Import field 'T1' is mapped to WeeWX field 'extraTemp1' but the
         import field 'T1' could not be found in one or more records.
         WeeWX field 'extraTemp1' will be set to 'None' in these records.
Warning: Import field 'T2' is mapped to WeeWX field 'extraTemp2' but the
         import field 'T2' could not be found in one or more records.
         WeeWX field 'extraTemp2' will be set to 'None' in these records.
Warning: Import field 'T3' is mapped to WeeWX field 'extraTemp3' but the
         import field 'T3' could not be found in one or more records.
         WeeWX field 'extraTemp3' will be set to 'None' in these records.
Warning: Import field 'H1' is mapped to WeeWX field 'extraHumid1' but the
         import field 'H1' could not be found in one or more records.
         WeeWX field 'extraHumid1' will be set to 'None' in these records.
Warning: Import field 'H2' is mapped to WeeWX field 'extraHumid2' but the
         import field 'H2' could not be found in one or more records.
         WeeWX field 'extraHumid2' will be set to 'None' in these records.
Warning: Import field 'Sm1' is mapped to WeeWX field 'soilMoist1' but the
         import field 'Sm1' could not be found in one or more records.
         WeeWX field 'soilMoist1' will be set to 'None' in these records.
Warning: Import field 'Sm2' is mapped to WeeWX field 'soilMoist2' but the
         import field 'Sm2' could not be found in one or more records.
         WeeWX field 'soilMoist2' will be set to 'None' in these records.
Warning: Import field 'Sm3' is mapped to WeeWX field 'soilMoist3' but the
         import field 'Sm3' could not be found in one or more records.
         WeeWX field 'soilMoist3' will be set to 'None' in these records.
Warning: Import field 'Sm4' is mapped to WeeWX field 'soilMoist4' but the
         import field 'Sm4' could not be found in one or more records.
         WeeWX field 'soilMoist4' will be set to 'None' in these records.
Warning: Import field 'Lw1' is mapped to WeeWX field 'leafWet1' but the
         import field 'Lw1' could not be found in one or more records.
         WeeWX field 'leafWet1' will be set to 'None' in these records.
Warning: Import field 'Lw2' is mapped to WeeWX field 'leafWet2' but the
         import field 'Lw2' could not be found in one or more records.
         WeeWX field 'leafWet2' will be set to 'None' in these records.
Warning: Import field 'St1' is mapped to WeeWX field 'soilTemp1' but the
         import field 'St1' could not be found in one or more records.
         WeeWX field 'soilTemp1' will be set to 'None' in these records.
Warning: Import field 'St2' is mapped to WeeWX field 'soilTemp2' but the
         import field 'St2' could not be found in one or more records.
         WeeWX field 'soilTemp2' will be set to 'None' in these records.
Warning: Import field 'St3' is mapped to WeeWX field 'soilTemp3' but the
         import field 'St3' could not be found in one or more records.
         WeeWX field 'soilTemp3' will be set to 'None' in these records.
Warning: Import field 'St4' is mapped to WeeWX field 'soilTemp4' but the
         import field 'St4' could not be found in one or more records.
         WeeWX field 'soilTemp4' will be set to 'None' in these records.
Warning: Import field 'Lt1' is mapped to WeeWX field 'leafTemp1' but the
         import field 'Lt1' could not be found in one or more records.
         WeeWX field 'leafTemp1' will be set to 'None' in these records.
Warning: Import field 'Lt2' is mapped to WeeWX field 'leafTemp2' but the
         import field 'Lt2' could not be found in one or more records.
         WeeWX field 'leafTemp2' will be set to 'None' in these records.
Unique records processed: 38284; Last timestamp: 2016-11-30 23:59:00 AEST (1480514340)

... (identical entries for periods 3 to 7 omitted for conciseness)

Period 8 ...
Warning: Import field 'T1' is mapped to WeeWX field 'extraTemp1' but the
         import field 'T1' could not be found in one or more records.
         WeeWX field 'extraTemp1' will be set to 'None' in these records.
Warning: Import field 'T2' is mapped to WeeWX field 'extraTemp2' but the
         import field 'T2' could not be found in one or more records.
         WeeWX field 'extraTemp2' will be set to 'None' in these records.
Warning: Import field 'T3' is mapped to WeeWX field 'extraTemp3' but the
         import field 'T3' could not be found in one or more records.
         WeeWX field 'extraTemp3' will be set to 'None' in these records.
Warning: Import field 'H1' is mapped to WeeWX field 'extraHumid1' but the
         import field 'H1' could not be found in one or more records.
         WeeWX field 'extraHumid1' will be set to 'None' in these records.
Warning: Import field 'H2' is mapped to WeeWX field 'extraHumid2' but the
         import field 'H2' could not be found in one or more records.
         WeeWX field 'extraHumid2' will be set to 'None' in these records.
Warning: Import field 'Sm1' is mapped to WeeWX field 'soilMoist1' but the
         import field 'Sm1' could not be found in one or more records.
         WeeWX field 'soilMoist1' will be set to 'None' in these records.
Warning: Import field 'Sm2' is mapped to WeeWX field 'soilMoist2' but the
         import field 'Sm2' could not be found in one or more records.
         WeeWX field 'soilMoist2' will be set to 'None' in these records.
Warning: Import field 'Sm3' is mapped to WeeWX field 'soilMoist3' but the
         import field 'Sm3' could not be found in one or more records.
         WeeWX field 'soilMoist3' will be set to 'None' in these records.
Warning: Import field 'Sm4' is mapped to WeeWX field 'soilMoist4' but the
         import field 'Sm4' could not be found in one or more records.
         WeeWX field 'soilMoist4' will be set to 'None' in these records.
Warning: Import field 'Lw1' is mapped to WeeWX field 'leafWet1' but the
         import field 'Lw1' could not be found in one or more records.
         WeeWX field 'leafWet1' will be set to 'None' in these records.
Warning: Import field 'Lw2' is mapped to WeeWX field 'leafWet2' but the
         import field 'Lw2' could not be found in one or more records.
         WeeWX field 'leafWet2' will be set to 'None' in these records.
Warning: Import field 'St1' is mapped to WeeWX field 'soilTemp1' but the
         import field 'St1' could not be found in one or more records.
         WeeWX field 'soilTemp1' will be set to 'None' in these records.
Warning: Import field 'St2' is mapped to WeeWX field 'soilTemp2' but the
         import field 'St2' could not be found in one or more records.
         WeeWX field 'soilTemp2' will be set to 'None' in these records.
Warning: Import field 'St3' is mapped to WeeWX field 'soilTemp3' but the
         import field 'St3' could not be found in one or more records.
         WeeWX field 'soilTemp3' will be set to 'None' in these records.
Warning: Import field 'St4' is mapped to WeeWX field 'soilTemp4' but the
         import field 'St4' could not be found in one or more records.
         WeeWX field 'soilTemp4' will be set to 'None' in these records.
Warning: Import field 'Lt1' is mapped to WeeWX field 'leafTemp1' but the
         import field 'Lt1' could not be found in one or more records.
         WeeWX field 'leafTemp1' will be set to 'None' in these records.
Warning: Import field 'Lt2' is mapped to WeeWX field 'leafTemp2' but the
         import field 'Lt2' could not be found in one or more records.
         WeeWX field 'leafTemp2' will be set to 'None' in these records.
Unique records processed: 38284; Last timestamp: 2017-06-30 23:59:00 AEST (1498831140)
Finished dry run import
308670 records were processed and 308670 unique records would have been imported.

In this case warnings are evident for numerous import/WeeWX field pairs that are mapped but for which no data could be found. If the warnings relate to fields that are not included in the import source data the warning may be safely ignored. If the warning relate to fields that the user expects to be in the import source data the issue should be investigated further before the import is completed

Once the dry run results are satisfactory the data can be imported using the following command:

wee_import --import-config=/var/tmp/wcat.conf --suppress-warnings

This will result in a preamble similar to that of a dry run. At the end of the preamble there will be a prompt:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
WeatherCat monthly .cat files in the '/var/tmp/wcat' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?

If there is more than one WeatherCat monthly .cat file then wee_import will provide summary information on a per period basis during the import. In addition, if the --date option is used then source data that falls outside the date or date range specified with the --date option is ignored. In such cases the preamble may look similar to:

Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
WeatherCat monthly .cat files in the '/var/tmp/wcat' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Period 1 - no records identified for import.
Period 2 ...
Period 2 - no records identified for import.
Period 3 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?

If the import parameters are acceptable enter y to proceed with the import or n to abort the import. If the import is confirmed, the source data will be imported, processed and saved in the WeeWX database. Information on the progress of the import will be displayed similar to the following:
```
Unique records processed: 2305; Last timestamp: 2016-12-30 00:00:00 AEST (1483020000)
```
Again if there is more than one WeatherCat monthly .cat file and if the --date option is used then the progress information may instead look similar to:
```
Period 4 ...
Unique records processed: 8908; Last timestamp: 2017-01-31 23:59:00 AEST (1485870900)
Period 5 ...
Unique records processed: 8029; Last timestamp: 2017-02-28 23:59:00 AEST (1488290100)
Period 6 ...
Unique records processed: 8744; Last timestamp: 2017-03-31 23:59:00 AEST (1490968500)
```
Note
Any periods for which no data could be obtained will be skipped. The lack of data may be due to a missing WeatherCat monthly .cat file. A short explanatory note to this effect will be displayed against the period concerned and an entry included in the log.

The line commencing with Unique records processed should update as records are imported with progress information on number of records processed, number of unique records imported and the date time of the latest record processed. If the import spans multiple months (ie multiple monthly .cat files) then a new Period line is created for each month.

Once the initial import is complete wee_import will, if requested, calculate any missing derived observations and rebuild the daily summaries. A brief summary should be displayed similar to the following:
```
Calculating missing derived observations ...
Processing record: 77782; Last record: 2017-06-30 00:00:00 AEST (1519826400)
Recalculating daily summaries...
Records processed: 77000; Last date: 2017-06-28 11:45:00 AEST (1519811100)
Finished recalculating daily summaries
Finished calculating missing derived observations
```
When the import is complete a brief summary is displayed similar to the following:
```
Finished import
308670 records were processed and 308670 unique records imported in 1907.61 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
```
Whilst wee_import will advise of the number of records processed and the number of unique records found, wee_import does know how many, if any, of the imported records were successfully saved to the database. You should look carefully through the WeeWX log file covering the wee_import session and take note of any records that were not imported. The most common reason for imported records not being saved to the database is because a record with that timestamp already exists in the database, in such cases something similar to the following will be found in the log:
```
Aug 22 14:38:28 stretch12 weewx[863]: manager: unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
```
In such cases take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.

Dealing with import failures

Sometimes bad things happen during an import.

If errors were encountered, or if you suspect that the WeeWX database has been contaminated with incorrect data, here are some things you can try to fix things up.

Manually delete the contaminated data. Use SQL commands to manipulate the data in the WeeWX archive database. The simplicity of this process will depend on your ability to use SQL, the amount of data imported, and whether the imported data was dispersed amongst existing. Once contaminated data have been removed the daily summary tables will need to be rebuilt using the wee_database utility.
Delete the database and start over. For SQLite, simply delete the database file. For MySQL, drop the database. Then try the import again.
Warning!
Deleting the database file or dropping the database will result in all data in the database being lost.
If the above steps are not appropriate then the database should be restored from backup. You did make a backup before starting the import?

wee_reports

In normal operation, WeeWX generates reports on each archive interval, when new data arrive. The reports utility is used to generate reports on demand. It uses the same configuration file that WeeWX uses.

Run the utility with the --help option to see how it is used:

wee_reports --help

This results in something like this:

Usage: wee_reports: [config_file] [timestamp] [--config=CONFIG_FILE] [--help]

Run all reports defined in the specified configuration file. Use this utility
to run reports immediately instead of waiting for the end of an archive
interval.

Options:
  -h, --help            show this help message and exit
  --config=CONFIG_FILE  Use the configuration file CONFIG_FILE

weewxd

The weewxd application is the heart of WeeWX. It can be run directly, or in the background as a daemon.

Run with the --help option to see how it is used:

weewxd --help

This results in output something like:

Usage: weewxd --help
       weewxd --version
       weewxd config_file [--daemon] [--pidfile=PIDFILE] 
                          [--exit]   [--loop-on-init]
                          [--log-label=LABEL]
           
  Entry point to the weewx weather program. Can be run directly, or as a daemon
  by specifying the '--daemon' option.

Arguments:
    config_file: The weewx configuration file to be used.


Options:
  -h, --help            show this help message and exit
  -d, --daemon          Run as a daemon
  -p PIDFILE, --pidfile=PIDFILE
                        Store the process ID in PIDFILE
  -v, --version         Display version number then exit
  -x, --exit            Exit on I/O and database errors instead of restarting
  -r, --loop-on-init    Retry forever if device is not ready on startup
  -n LABEL, --log-label=LABEL
                        Label to use in syslog entries

wunderfixer

For a number of reasons, posting weather observation data to Weather Underground often results in missing observations. The wunderfixer utility can re-post such missing observations. This section explains how to do it.

Note
wunderfixer uses the Weather Underground convention of what is a day. That is, a day runs from the record timestamped at midnight to the last record timestamped on the same day. By this convention, the first record is actually an archive for the last archive interval of the previous day.

Before starting, it's worth running the utility with the --help flag to see how wunderfixer is used:

wunderfixer --help

This will result in an output that looks something like this:

Usage: wunderfixer CONFIG_FILE|--config=CONFIG_FILE
                  [--binding=BINDING]
                  [--station=STATION] [--password=PASSWORD] [--api-key=API_KEY]
                  [--date=YYYY-mm-dd] [--epsilon=SECONDS] [--upload-only=SECONDS]
                  [--verbose] [--test] [--query] [--timeout=SECONDS]
                  [--help]

This utility fills in missing data on the Weather Underground.  It goes through
all the records in a weewx archive for a given day, comparing to see whether a
corresponding record exists on the Weather Underground. If not, it will publish
a new record on the Weather Underground with the missing data.

Be sure to use the --test switch first to see whether you like what it
proposes!

Options:
  -h, --help            show this help message and exit
  -c CONFIG_PATH, --config=CONFIG_PATH
                        Use configuration file CONFIG_PATH. Default is
                        /etc/weewx/weewx.conf or /home/weewx/weewx.conf.
  -b BINDING, --binding=BINDING
                        The database binding to be used. Default is
                        'wx_binding'.
  -s STATION, --station=STATION
                        Weather Underground station to check. Optional.
                        Default is to take from configuration file.
  -p PASSWORD, --password=PASSWORD
                        Weather Underground station password. Optional.
                        Default is to take from configuration file.
  -k API_KEY, --api-key=API_KEY
                        Weather Underground API key. Optional. Default is to
                        take from configuration file.
  -d YYYY-mm-dd, --date=YYYY-mm-dd
                        Date to check as a string of form YYYY-mm-dd. Default
                        is today.
  -e SECONDS, --epsilon=SECONDS
                        Timestamps within this value in seconds compare true.
                        Default is 120.
  -u SECONDS, --upload-only=SECONDS
                        Upload only records every SECONDS apart or more.
                        Default is 300.
  -v, --verbose         Print useful extra output.
  -l LOG_FACILITY, --log=LOG_FACILITY
                        OBSOLETE. Logging will always occur.
  -t, --test            Test what would happen, but don't do anything.
  -q, --query           For each record, query the user before making a
                        change.
  -o SECONDS, --timeout=SECONDS
                        Socket timeout in seconds. Default is 10.

Options 'station', 'password', and 'api-key' must be supplied either on the
command line, or in the configuration file.

Actions and options

The wunderfixer options are described in more detail below:

Option --config

The utility is pretty good at "guessing" where the weewx configuration file is, but if you've done an unusual install, you may have to tell it explicitly. You can do this by using the --config option:

wunderfixer --config=/this/directory/weewx.conf

Option --binding

Specifies the data binding to be used as the source of data for missing records to be published to Weather Underground. Default is wx_binding. The --binding option is used as follows:

wunderfixer --binding=another_binding

Option --station

The weather station ID of the Weather Underground PWS. The default is the station specified in the [StdRESTful][[Wunderground]] section of the configuration file. The --station option is used as follows:

wunderfixer --station=AB123456789

Option --password

The password corresponding to the weather station ID. The default is the password specified in the [StdRESTful][[Wunderground]] section of the configuration file. The --password option is used as follows:

wunderfixer --station=AB123456789 --password=hardtoguess

Option --api-key

Your API key from the Weather Underground. You can find (or generate, if you do not have one) your API key at https://www.wunderground.com/member/api-keys. The default is the value for option api_key specified in the [StdRESTful][[Wunderground]] section of the configuration file. The --api-key option is used as follows:

wunderfixer --station=AB123456789 --api-key=04255779e9aa45b6e4579938630e67b

Option --date

By default wunderfixer checks the current date according to the system date-time. Use this option to specify a different date. This feature is useful when running as a cron job. The --date option accepts dates in the format YYYY-mm-dd. For example:

wunderfixer --date=2016-04-20

Option --epsilon

At times Weather Underground records may have a date-time that is slightly different to the timestamp of the record as recorded by WeeWX. The --epsilon option allows timestamps that are within the --epsilon value seconds of each other to be considered the same. The default value is 120. The --epsilon option is used as follows:

wunderfixer --epsilon=60

Option --upload-only

The Weather Underground will store records no more often than every 5 minutes. If your archive interval is something less than this (for example, 1 minute), then wunderfixer will report many "missing" records. Trying to upload them would be futile and take unnecessary bandwidth, because the WU will just reject them. For this reason, by default, wunderfixer will upload records no more frequently than every 300 seconds (5 minutes). This option allows you to override this. The option is used as follows:

wunderfixer --upload-only=60

If your archive interval is 60 seconds, this will cause wunderfixer to upload every record, instead of every 5th record. NB: unfortunately, the WU will not store them.

Option --verbose

Use of the --verbose option results in wunderfixer displaying useful additional information during execution. The --verbose option is of little use when wunderfixer is run as a cron job.

Option --log

The --log option is obsolete and no longer has any affect, but is retained for backwards compatibility.

Option --test

The --test option will cause wunderfixer to do everything except upload any missing data to Weather Underground. Summary information on any identified missing data will be displayed.

Option --query

The --query option will cause wunderfixer to seek user confirmation before each missing record is uploaded to Weather Underground. When queried you may respond with y to publish the record, n to skip the record without publishing, a to publish the record and automatically publish all further records or q to skip the record and quit wunderfixer. The --query option should not be used as part of a cron job.

Option --timeout

Sets how long to wait for a response from the Weather Underground. At times of high usage, it might help to increase this value. Default is 10 seconds.

wunderfixer --timeout=20

Running directly

Run wunderfixer directly to publish missing data for a single day. This may be useful if you wish to restore data from a short network outage or similar problems that resulted in missing Weather Underground data. Running wunderfixer directly is also a useful exercise in the lead up to running as a cron job, as it allows you to verify correct operation before automating the process.

The following instructions will publish data for the current day for the station whose credentials appear in the [StdRESTful][[Wunderground]] section of weewx.conf.

Note
Data for another station or date could be published by using the --station, --password and --date options.

Note
The data fields shown in the example outputs below are indicative only and the actual output may or may not include other fields.

To run wunderfixer directly:

Before publishing missing data for the first time, do a test run by specifying the --test option.

wunderfixer --test

This will result in output similar to the following:

Using configuration file /home/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
2016-09-22 06:30:00 AEST (1474489800); 29.920";  58.9F;  79%; 1.0 mph; 248 deg; 6.0 mph gust;  52.4F; 0.00" rain; 0.01" ... skipped.
2016-09-22 07:35:00 AEST (1474493700); 29.931";  64.9F;  65%; 2.0 mph; 180 deg; 7.0 mph gust;  52.8F; 0.00" rain; 0.01" ... skipped.
2016-09-22 07:55:00 AEST (1474494900); 29.934";  65.8F;  63%; 2.0 mph; 180 deg;10.0 mph gust;  52.8F; 0.00" rain; 0.01" ... skipped.
2016-09-22 08:20:00 AEST (1474496400); 29.938";  66.5F;  59%; 5.0 mph; 180 deg;12.0 mph gust;  51.7F; 0.00" rain; 0.01" ... skipped.

This output indicates that four records were found to be missing. The word 'skipped' at the end of each line indicates that whilst wunderfixer detected the record as missing, the record was skipped and not published to Weather Underground. If no missing records were found then no records will be listed.

Note
Use of the --verbose option will display additional information including the station, the date, and the number of records.

Once the test results are satisfactory, publish the missing records using the command:

wunderfixer

This will result in output similar to the following:

Using configuration file /home/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
2016-09-22 06:30:00 AEST (1474489800); 29.920";  58.9F;  79%; 1.0 mph; 248 deg; 6.0 mph gust;  52.4F; 0.00" rain; 0.01" ... published.
2016-09-22 07:35:00 AEST (1474493700); 29.931";  64.9F;  65%; 2.0 mph; 180 deg; 7.0 mph gust;  52.8F; 0.00" rain; 0.01" ... published.
2016-09-22 07:55:00 AEST (1474494900); 29.934";  65.8F;  63%; 2.0 mph; 180 deg;10.0 mph gust;  52.8F; 0.00" rain; 0.01" ... published.
2016-09-22 08:20:00 AEST (1474496400); 29.938";  66.5F;  59%; 5.0 mph; 180 deg;12.0 mph gust;  51.7F; 0.00" rain; 0.01" ... published.

This output indicates that four records were found to be missing. This time word 'skipped' at the end of each line has been replaced with the word 'published' indicating that wunderfixer has published the record to Weather Underground. If no missing records were found then no records will be listed.

If the --query option had been used the output would be something like this:

Using configuration file /home/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station:   ABCDEFGH123
Date to check:                 2016-09-22
Number of archive records:     288
Number of WU records:          284
Number of missing records:     4

Missing records:
2016-09-22 06:30:00 AEST (1474489800); 29.920";  58.9F;  79%; 1.0 mph; 248 deg; 6.0 mph gust;  52.4F; 0.00" rain; 0.01" ...fix? (y/n/a/q):y
 ...published.
2016-09-22 07:35:00 AEST (1474493700); 29.931";  64.9F;  65%; 2.0 mph; 180 deg; 7.0 mph gust;  52.8F; 0.00" rain; 0.01" ...fix? (y/n/a/q):y
 ...published.
2016-09-22 07:55:00 AEST (1474494900); 29.934";  65.8F;  63%; 2.0 mph; 180 deg;10.0 mph gust;  52.8F; 0.00" rain; 0.01" ...fix? (y/n/a/q):y
 ...published.
2016-09-22 08:20:00 AEST (1474496400); 29.938";  66.5F;  59%; 5.0 mph; 180 deg;12.0 mph gust;  51.7F; 0.00" rain; 0.01" ...fix? (y/n/a/q):

Note
It is possible that two wunderfixer sessions for the same station and date may return two different sets of missing records. This is a vagary of Weather Underground and is not a cause for concern. If you are concerned that not all missing records were published you can run wunderfixer again with the same settings. Re-publishing records that already exist on Weather Underground has no known adverse effects.

Running as a cron job

At times Weather Underground flatly refuses to update older dates, typically those more than a couple of weeks old. For this reason running wunderfixer using a nightly cron job is advisable. To run wunderfixer as a nightly cron job:

Ensure that wunderfixer operates as intended when run directly.
Note
Beware that running wunderfixer with the --test option does not utilise the station password. So be sure to run at least once without the --test option to ensure that the correct station and password parameters have been specified.
Decide how often and when to run wunderfixer. Given the use of the midnight to midnight day, it may be prudent to run wunderfixer close to midnight at the end of the day after the second last archive record of the day (remember the last archive record of the day is at midnight). For an installation that uses a five minute archive period this would be between 11:55pm and midnight. Furthermore, as Weather Underground at times does not update a record despite returning 'success' it may be advisable to run wunderfixer more than once per day. For the purposes of these instructions we will set a cron job to run wunderfixer at 11:58pm daily.

Create (if one does not exist) or edit a crontab file using the following command:

crontab -e

Note
The above command will create a crontab file for the currently logged in user. Any commands in the crontab will be executed as the user who owns the crontab. If the currently logged in user does not have adequate permissions to run wunderfixer you may need to use the root user crontab by using the command sudo crontab -e.

This should display a crontab file similar to the following:

# Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command

To run wunderfixer at 11:58pm daily insert the following lines at the end of the crontab file:
```
# Run wunderfixer at 1158pm daily
58 23 * * * /home/weewx/bin/wunderfixer --log weewx > /dev/null 2>&1
```
The above command executes wunderfixer each night at 11:58pm and publishes any missing records for the current day for the station whose credentials appear in the [StdRESTful][[Wunderground]] section of weewx.conf. wunderfixer log output will be sent to the log file used by WeeWX. The > /dev/null 2>&1 portion of the command suppresses any wunderfixer screen output by redirecting any screen and error output to the null device.
Save the crontab file and exit the editor.
wunderfixer will now be run at 11:58pm daily. To run wunderfixer again at a different time either create a new crontab entry or modify the scheduling of the existing entry to suit.
Verify that wunderfixer is being executed by cron as expected:
- Check the cron log to verify that wunderfixer was run at the expected time. The location of the cron log will vary depending on your version of Linux.
- Check the wunderfixer log to confirm how many, if any, records were uploaded to Weather Underground. The location of the wunderfixer log will depend on the --log option used and your version of Linux.
- Check the Weather Underground Weather History for the station concerned to ensure that any missing data was accepted by Weather Underground.

Dealing with failures

Weather Underground is known for a number of quirks in behaviour, this coupled with a misconfigured wunderfixer session or incomplete WeeWX archive data can lead to unexpected results when using wunderfixer. Some of the issues and errors that may be encountered and possible solutions are detailed below.

Updated records do not appear on Weather Underground

What appears to be a successfully posted record to Weather Underground may not appear in the station history for a number of reasons. Some of these reasons include:

Weather Underground just chose to ignore the post. In this case, use wunderfixer to publish the missing record again.
Weather Underground has not yet updated the stations history. Sometimes Weather Underground takes a short time to update the stations history. If the record does not appear in the station history within the next hour use wunderfixer to publish the missing record again.
There is no corresponding record in the WeeWX archive. Check the WeeWX archive to see if the record concerned exists. If the record does not exist then it must be re-created before it can be published by wunderfixer.
wunderfixer may have been run with the --test option thus only checking for missing records and not publishing them. In this case, run wunderfixer again without the --test option.

Incorrect data was published on Weather Underground

If you have published incorrect data to Weather Underground there is little that can be done other than manually deleting the erroneous observation data from Weather Underground. Weather Underground does allow observation data to be edited; however, the only available option is to delete the observation(s); it is not possible to change the value recorded against an observation to the correct value nor is it possible to delete an entire record. It appears that when all observations for a given record are deleted Weather Underground retains the record but with no observation data. Consequently, subsequent wunderfixer sessions using the correct data will not result in the correct data being published as Weather Underground allows publishing of new records but not re-publishing of existing records. For these reasons you should ensure wunderfixer is operating correctly with the --test option before publishing any missing data or running wunderfixer as a cron job.