Running WeeWX during a
weectl import session can lead to abnormal
termination of the import. If WeeWX must remain running (e.g., so that
live data is not lost) run the
weectl import session on another machine
or to a second database and merge the in-use and second database once the
import is complete.
weectl import can import data from a single CSV file. The CSV source file
must be structured as follows:
The file must have a header row consisting of a comma separated list of field names. The field names can be any valid string as long as each field name is unique within the list. There is no requirement for the field names to be in any particular order as long as the same order is used for the observations on each row in the file. These field names will be mapped to WeeWX field names in the
[CSV]section of the import configuration file.
Observation data for a given date-time must be listed on a single line with individual fields separated by a comma. The fields must be in the same order as the field names in the header row.
Blank fields are represented by the use of white space or no space only between commas.
Direction data being imported may be represented as numeric degrees or as a string representing the cardinal, intercardinal and/or secondary intercardinal directions.
There must a field that represents the date-time of the observations on each line. This date-time field must be either a Unix epoch timestamp or any date-time format that can be represented using Python strptime() format codes.
A CSV file suitable for import by
weectl import may look like this:
28/11/2017 08:00:00,1016.9,24.6,84,1.8,113,8,0,359,3.8,"start of observations"
28/11/2017 08:10:00,1016.9,25.4,80,4.4,127,11.3,0,787,5.1,"note temperature"
28/11/2017 08:25:00,1017,25.5,78,2.9,48,9.7,0,303,3.4,"forecast received"
Cardinal, intercardinal and/or secondary intercardinal directions may be represented by one, two or three letter abbreviations e.g., N, SE or SSW; by a single word e.g., North, Southwest or Southsouthwest or by hyphenated or spaced words e.g., North West or South-south-west. Capitalisation is ignored as are any spaces, hyphens or other white space. At present only English abbreviations and directions are supported.
Mapping data to archive fields¶
The WeeWX archive fields populated during a CSV import depend on the
CSV-to-WeeWX field mappings specified in
[[FieldMap]] stanza in the import
configuration file. If a valid field mapping exists, the WeeWX field exists
in the WeeWX archive table schema and provided the mapped CSV field contains
valid data, the corresponding WeeWX field will be populated.
The use of the calc_missing option in the import configuration file may result in a number of derived fields being calculated from the imported data. If these derived fields exist in the in-use database schema they will be saved to the database as well.
To import observations from a CSV file:
Ensure the source data file is in a directory accessible by the machine that will run
weectl import. For the purposes of the following examples the source data file
data.csvlocated in the
/var/tmpdirectory will be used.
Make a backup of the WeeWX database in case the import should go awry.
Create an import configuration file. In this case we will make a copy of the example CSV import configuration file and save it as
cp /home/weewx/util/import/csv-example.conf /var/tmp/csv.conf
Confirm that the
sourceoption is set to CSV:
source = CSV
Confirm the following options in the
[CSV]section are set:
file. The full path and file name of the file containing the CSV formatted data to be imported.
delimiter. The single character used to separate fields.
interval. Determines how the WeeWX interval field is derived.
qc. Determines whether quality control checks are performed on the imported data.
calc_missing. Determines whether missing derived observations will be calculated from the imported data.
ignore_invalid_data. Determines whether invalid data in a source field is ignored or the import aborted.
tranche. The number of records written to the WeeWX database in each transaction.
UV_sensor. Whether a UV sensor was installed when the source data was produced.
solar_sensor. Whether a solar radiation sensor was installed when the source data was produced.
raw_datetime_format. The format of the imported date time field.
rain. Determines how the WeeWX rain field is derived.
wind_direction. Determines how imported wind direction fields are interpreted.
[[FieldMap]]. Defines the mapping between imported data fields and WeeWX archive fields. Also defines the units of measure for each imported field.
When first importing data it is prudent to do a dry run import before any data are actually imported. A dry run import will perform all steps of the import without actually writing imported data to the WeeWX database. In addition, consideration should be given to any additional options such as
To perform a dry run enter the following command:
weectl import --import-config=/var/tmp/csv.conf --dry-run
The output should be something like:
Using WeeWX configuration file /home/weewx/www-data/weewx.conf Starting weectl import... A CSV import from source file '/var/tmp/data.csv' has been requested. Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. This is a dry run, imported data will not be saved to archive. Starting dry run import ... 27337 records identified for import. Unique records processed: 27337; Last timestamp: 2018-03-03 06:00:00 AEST (1520020800) Finished dry run import 27337 records were processed and 27337 unique records would have been imported.
The output includes details about the data source, the destination of the imported data and some other details on how the data will be processed. The import will then be performed but no data will be written to the WeeWX database. Upon completion a brief summary of the records processed is provided.
Once the dry run results are satisfactory the data can be imported using the following command:
weectl import --import-config=/var/tmp/csv.conf
This will result in a short preamble similar to that from the dry run. At the end of the preamble there will be a prompt:
Using WeeWX configuration file /home/weewx/www-data/weewx.conf Starting weectl import... A CSV import from source file '/var/tmp/data.csv' has been requested. Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. Starting import ... 27337 records identified for import. Proceeding will save all imported records in the WeeWX archive. Are you sure you want to proceed (y/n)?
If the import parameters are acceptable enter
yto proceed with the import or
nto abort the import. If the import is confirmed the source data will be imported, processed and saved in the WeeWX database. Information on the progress of the import will be displayed similar to the following:
Unique records processed: 3250; Last timestamp: 2017-12-09 14:45:00 AEST (1512794700)
The line commencing with
Unique records processedshould update as records are imported with progress information on number of records processed, number of unique records imported and the date time of the latest record processed. Once the initial import is complete
weectl importwill, if requested, calculate any missing derived observations and rebuild the daily summaries. A brief summary should be displayed similar to the following:
Calculating missing derived observations... Processing record: 27337; Last record: 2018-03-03 06:00:00 AEST (1520020800) Recalculating daily summaries... Records processed: 27337; Last date: 2018-03-03 06:00:00 AEST (1520020800) Finished recalculating daily summaries Finished calculating missing derived observations
When the import is complete a brief summary is displayed similar to the following:
Finished import 27337 records were processed and 27337 unique records imported in 113.91 seconds. Those records with a timestamp already in the archive will not have been imported. Confirm successful import in the WeeWX log file.
weectl importwill advise of the number of records processed and the number of unique records found,
weectl importdoes know how many, if any, of the imported records were successfully saved to the database. You should look carefully through the WeeWX log file covering the
weectl importsession and take note of any records that were not imported. The most common reason for imported records not being saved to the database is because a record with that timestamp already exists in the database, in such cases something similar to the following will be found in the log:
2023-11-04 15:33:01 weectl-import: ERROR weewx.manager: Unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
In such cases you should take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.