CSV

Warning

Running WeeWX during a weectl import session can lead to abnormal termination of the import. If WeeWX must remain running (e.g., so that live data is not lost) run the weectl import session on another machine or to a second database and merge the in-use and second database once the import is complete.

weectl import can import data from a single CSV file. The CSV source file must be structured as follows:

The file must have a header row consisting of a comma separated list of field names. The field names can be any valid string as long as each field name is unique within the list. There is no requirement for the field names to be in any particular order as long as the same order is used for the observations on each row in the file. These field names will be mapped to WeeWX field names in the [CSV] section of the import configuration file.
Observation data for a given date-time must be listed on a single line with individual fields separated by a comma. The fields must be in the same order as the field names in the header row.
Blank fields are represented by the use of white space or no space only between commas.
Direction data being imported may be represented as numeric degrees or as a string representing the cardinal, intercardinal and/or secondary intercardinal directions.
There must a field that represents the date-time of the observations on each line. This date-time field must be either a Unix epoch timestamp or any date-time format that can be represented using Python strptime() format codes.

A CSV file suitable for import by weectl import may look like this:

Time,Barometer,Temp,Humidity,Windspeed,Dir,Gust,Dayrain,Radiation,Uv,Comment
28/11/2017 08:00:00,1016.9,24.6,84,1.8,113,8,0,359,3.8,"start of observations"
28/11/2017 08:05:00,1016.9,25.1,82,4.8,135,11.3,0,775,4.7,
28/11/2017 08:10:00,1016.9,25.4,80,4.4,127,11.3,0,787,5.1,"note temperature"
28/11/2017 08:15:00,1017,25.7,79,3.5,74,11.3,0,800,5.4,
28/11/2017 08:20:00,1016.9,25.9,79,1.6,95,9.7,0,774,5.5,
28/11/2017 08:25:00,1017,25.5,78,2.9,48,9.7,0,303,3.4,"forecast received"
28/11/2017 08:30:00,1017.1,25.1,80,3.1,54,9.7,0,190,3.6,

or this:

Time,Barometer,Temp,Humidity,Windspeed,Dir,Gust,Dayrain,Radiation,Uv
2/1/2017 06:20:00,1006.4,4.8,48,2.8,NE,4,0,349,2.8
2/1/2017 06:25:00,1006.9,5.0,48,3.8,NNE,21.3,0,885,4.3
2/1/2017 06:30:00,1006.8,5.4,47,3.4,North,12.3,0,887,5.3
2/1/2017 06:35:00,1007,5.2,49,5.5,NNE,13.3,0,600,5.4
2/1/2017 06:40:00,1006.9,5.7,49,2.6,ESE,9.7,0,732,5.5
2/1/2017 06:45:00,1007,5.5,48,1.9,Southsoutheast,9.8,0,393,6.4
2/1/2017 06:50:00,1007.1,5.2,50,2.1,southeast,9.9,0,180,6.6

Note

Cardinal, intercardinal and/or secondary intercardinal directions may be represented by one, two or three letter abbreviations e.g., N, SE or SSW; by a single word e.g., North, Southwest or Southsouthwest or by hyphenated or spaced words e.g., North West or South-south-west. Capitalisation is ignored as are any spaces, hyphens or other white space. At present only English abbreviations and directions are supported.

Mapping data to archive fields¶

The WeeWX archive fields populated during a CSV import depend on the CSV-to-WeeWX field mappings specified in [[FieldMap]] stanza in the import configuration file. If a valid field mapping exists, the WeeWX field exists in the WeeWX archive table schema and provided the mapped CSV field contains valid data, the corresponding WeeWX field will be populated.

Note

The use of the calc_missing option in the import configuration file may result in a number of derived fields being calculated from the imported data. If these derived fields exist in the in-use database schema they will be saved to the database as well.

Step-by-step instructions¶

To import observations from a CSV file:

Ensure the source data file is in a directory accessible by the machine that will run weectl import. For the purposes of the following examples the source data file data.csv located in the /var/tmp directory will be used.
Make a backup of the WeeWX database in case the import should go awry.
Create an import configuration file. In this case we will make a copy of the example CSV import configuration file and save it as csv.conf in the /var/tmp directory:
```
cp /home/weewx/util/import/csv-example.conf /var/tmp/csv.conf
```
Confirm that the source option is set to CSV:
```
source = CSV
```
Confirm the following options in the [CSV] section are set:
- file. The full path and file name of the file containing the CSV formatted data to be imported.
- delimiter. The single character used to separate fields.
- interval. Determines how the WeeWX interval field is derived.
- qc. Determines whether quality control checks are performed on the imported data.
- calc_missing. Determines whether missing derived observations will be calculated from the imported data.
- ignore_invalid_data. Determines whether invalid data in a source field is ignored or the import aborted.
- tranche. The number of records written to the WeeWX database in each transaction.
- UV_sensor. Whether a UV sensor was installed when the source data was produced.
- solar_sensor. Whether a solar radiation sensor was installed when the source data was produced.
- raw_datetime_format. The format of the imported date time field.
- rain. Determines how the WeeWX rain field is derived.
- wind_direction. Determines how imported wind direction fields are interpreted.
- [[FieldMap]]. Defines the mapping between imported data fields and WeeWX archive fields. Also defines the units of measure for each imported field.

When first importing data it is prudent to do a dry run import before any data are actually imported. A dry run import will perform all steps of the import without actually writing imported data to the WeeWX database. In addition, consideration should be given to any additional options such as --date.

To perform a dry run enter the following command:

weectl import --import-config=/var/tmp/csv.conf --dry-run

The output should be something like:

Using WeeWX configuration file /home/weewx/www-data/weewx.conf
Starting weectl import...
A CSV import from source file '/var/tmp/data.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
27337 records identified for import.
Unique records processed: 27337; Last timestamp: 2018-03-03 06:00:00 AEST (1520020800)
Finished dry run import
27337 records were processed and 27337 unique records would have been imported.

The output includes details about the data source, the destination of the imported data and some other details on how the data will be processed. The import will then be performed but no data will be written to the WeeWX database. Upon completion a brief summary of the records processed is provided.

Once the dry run results are satisfactory the data can be imported using the following command:

weectl import --import-config=/var/tmp/csv.conf

This will result in a short preamble similar to that from the dry run. At the end of the preamble there will be a prompt:

Using WeeWX configuration file /home/weewx/www-data/weewx.conf
Starting weectl import...
A CSV import from source file '/var/tmp/data.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
27337 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?

If the import parameters are acceptable enter y to proceed with the import or n to abort the import. If the import is confirmed the source data will be imported, processed and saved in the WeeWX database. Information on the progress of the import will be displayed similar to the following:
```
Unique records processed: 3250; Last timestamp: 2017-12-09 14:45:00 AEST (1512794700)
```
The line commencing with Unique records processed should update as records are imported with progress information on number of records processed, number of unique records imported and the date time of the latest record processed. Once the initial import is complete weectl import will, if requested, calculate any missing derived observations and rebuild the daily summaries. A brief summary should be displayed similar to the following:
```
Calculating missing derived observations...
Processing record: 27337; Last record: 2018-03-03 06:00:00 AEST (1520020800)
Recalculating daily summaries...
Records processed: 27337; Last date: 2018-03-03 06:00:00 AEST (1520020800)
Finished recalculating daily summaries
Finished calculating missing derived observations
```
When the import is complete a brief summary is displayed similar to the following:
```
Finished import
27337 records were processed and 27337 unique records imported in 113.91 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
```
Whilst weectl import will advise of the number of records processed and the number of unique records found, weectl import does know how many, if any, of the imported records were successfully saved to the database. You should look carefully through the WeeWX log file covering the weectl import session and take note of any records that were not imported. The most common reason for imported records not being saved to the database is because a record with that timestamp already exists in the database, in such cases something similar to the following will be found in the log:
```
2023-11-04 15:33:01 weectl-import[3795]: ERROR weewx.manager: Unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
```
In such cases you should take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.