CANOE Electricity Sector — Data Processing Documentation

Comprehensive documentation of the data pipeline that converts upstream data sources into a Temoa-ready SQLite database for the Canadian Open Energy (CANOE) model.

Table of Contents

1. Overview
2. Pipeline Architecture
3. Configuration and Setup
4. Pre-Processing
5. Provincial Grid Infrastructure
6. Generator Aggregation
7. Capacity Factors
8. Capacity Credits and Reserve Margin
9. New Wind and Solar Characterisation
10. Energy Storage
11. CCS Retrofits
12. Interfaces and Interties
13. Constraints
14. Currency Conversion
15. Post-Processing
16. Unit Conventions
17. Known Assumptions and Limitations

1. Overview

Purpose

The CANOE electricity sector aggregation tool automatically constructs a Temoa-compatible SQLite database representing the Canadian electricity system. It downloads data from several upstream databases — primarily the Canadian Open-Source Database for Energy Research and Systems-Modelling (CODERS) and the U.S. National Renewable Energy Laboratory (NREL) Annual Technology Baseline (ATB) — processes and harmonises that data, and writes it into a structured SQLite database suitable for long-term capacity expansion modelling.

What the Tool Produces

• A Temoa-schema SQLite database containing all technology, cost, capacity, demand, emissions, and time-series data required to run an electricity sector capacity expansion model.
• Optionally, an Excel workbook clone of the database for inspection and review.
• Diagnostic plots (PDF) showing demand profiles, capacity factors, capacity credits, and intertie flows for visual verification.

Scope

The model covers:

• Regions: Canadian provinces (configurable; defaults to all ten provinces).
• Technologies: Existing and new generators (thermal, nuclear, hydro, wind, solar), energy storage (batteries, pumped hydro), CCS retrofits, and transmission/distribution dummy technologies.
• Time resolution: Hourly (8,760 time-slices per year), with representative-day or full-year options.
• Planning horizon: Configurable five-year periods (default: 2025–2045).

2. Pipeline Architecture

The aggregation runs in a fixed order, orchestrated by electricity_sector.py:

1. setup.py            → Load configuration, download ATB workbook, build mappings
2. instantiate_database → Create or wipe SQLite database from schema
3. pre_processing.py   → Write time periods, regions, seasons, times-of-day
4. provincial_grids.py → Reserve margin, demand, transmission/distribution
5. generators.py       → Existing & new generators, storage, CCS retrofits
6. interfaces.py       → Interprovincial and international interties
7. post_processing.py  → Imports, commodity table, unused tech cleanup, references

Each step opens its own SQLite connection, writes data, commits, and closes. The database grows incrementally as each module runs.

Data Flow Diagram

                    ┌──────────────┐
                    │  CODERS API  │
                    └──────┬───────┘
                           │
     ┌─────────────────────┼──────────────────────┐
     │                     │                      │
     ▼                     ▼                      ▼
┌─────────┐        ┌──────────────┐       ┌──────────────┐
│Generators│        │ Provincial   │       │  Interface   │
│ Table    │        │ Demand (hrly)│       │  Flows (hrly)│
└────┬─────┘        └──────┬───────┘       └──────┬───────┘
     │                     │                      │
     ▼                     ▼                      ▼
┌─────────────────────────────────────────────────────────┐
│                 generators.py                           │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐              │
│  │ Existing │  │   New    │  │   CCS    │              │
│  │Generators│  │Generators│  │Retrofits │              │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘              │
│       │              │              │                    │
│       ▼              ▼              ▼                    │
│  ┌──────────────────────────────────────┐               │
│  │ Generic Data (costs, efficiency,     │               │
│  │ emissions, lifetime, ramp rates)     │               │
│  │ Source: ATB preferred, CODERS backup │               │
│  └──────────────────────────────────────┘               │
└──────────────────────────┬──────────────────────────────┘
                           │
     ┌─────────────────────┼──────────────────────┐
     │                     │                      │
     ▼                     ▼                      ▼
┌──────────┐       ┌──────────────┐       ┌──────────────┐
│ NREL ATB │       │ IESO Public  │       │ Renewables   │
│ CSV + WB │       │ Data (ON)    │       │ Ninja API    │
└────┬─────┘       └──────┬───────┘       └──────┬───────┘
     │                    │                      │
     ▼                    ▼                      ▼
┌─────────────────────────────────────────────────────────┐
│   Capacity Factors, Capacity Credits, Costs             │
│   → Written to Temoa SQLite database tables             │
└─────────────────────────────────────────────────────────┘

3. Configuration and Setup

3.1 Configuration File (params.yaml)

The primary configuration file controls all aspects of aggregation. Key parameters include:

Aggregation Switches

Boolean switches enable/disable major pipeline components:

3.2 Technology Configuration CSVs

Technologies are defined through several CSV files in input_files/:

• generator_technologies.csv: Maps generator types to CODERS/ATB equivalents, technology flags, I/O commodities, number of new-capacity bins, and whether to use ATB or CODERS data.
• storage_technologies.csv: Similar configuration for storage technologies (batteries, pumped hydro).
• transmission_technologies.csv: Defines dummy technologies for transmission (TX→DX), distribution (DX→DEM), demand, and interties.
• ccs_retrofit_technologies.csv: Defines CCS retrofit options with capture rates and ATB mappings.
• import_technologies.csv: Defines fuel import dummy technologies.
• commodities.csv: Maps commodity codes to names, flags (source/physical/demand/emissions), units, and descriptions.
• regions.csv: Defines provinces, their CODERS equivalents, and whether they are endogenous to the model.
• time.csv: Maps 8,760 hours to seasons (days) and times-of-day, defining the temporal resolution.
• units.csv: Conversion factors for all unit transformations (e.g., MW→GW, $/kW→M$/GW).

3.3 The config Singleton

On import, setup.py instantiates a singleton config object that:

1. Loads params.yaml and all CSV configuration files.
2. Builds mapping dictionaries:
3. gen_map: CODERS generator type → CANOE tech code
4. storage_map: (CODERS storage type, duration) → CANOE tech code
5. region_map: CODERS province name → CANOE region code
6. existing_map: CODERS generator type → CANOE existing-tech code
7. Downloads the NREL ATB master Excel workbook if not already cached.
8. Initialises per-region VRE generation arrays for capacity credit calculations.

3.4 Data Caching

All downloaded data is cached locally in data_cache/. The caching system:

• Checks if a local file exists before downloading.
• Stores CODERS API responses as CSV files with cleaned endpoint names.
• Tracks download dates in dates.csv for reference provenance.
• Converts XML and Excel downloads to CSV or pickle for faster subsequent loads.
• Can be forced to re-download with force_download: true.

3.5 Bibliography and Data IDs

The tool maintains a bibliography of all data sources, assigning each a unique ID (E01, E02, ...) which is stored in the DataSource table. Every data row in the database carries a data_id formed as {prefix}{region}{version} for traceability, and a data_source foreign key referencing the bibliography.

4. Pre-Processing

Module: pre_processing.py

Pre-processing populates the foundational Temoa tables that other modules depend on:

4.1 Time Periods

Future model periods from params.yaml are written to TimePeriod with flag "f" (future). An additional period is appended beyond the last model period (at last_period + period_step) to define the model horizon endpoint. Existing-capacity vintage periods are added later by generators.py with flag "e".

4.2 Regions

All endogenous model regions from regions.csv are written to the Region table with their descriptions.

4.3 Temporal Structure

The time.csv file defines the mapping of 8,760 hours to:

• Seasons (SeasonLabel): Each row defines a season label corresponding to a representative day or day-of-year.
• Times of day (TimeOfDay): 24 hourly labels (H01–H24) within each season.

These labels form the (season, tod) index used throughout all time-series data in the database.

Assumption: The temporal resolution is fixed at the hourly level within a 365-day year. Leap-year data (if present) is truncated to 8,760 hours.

5. Provincial Grid Infrastructure

Module: provincial_grids.py

This module handles three main tasks: demand, transmission structure, and planning reserve margin.

5.1 Electricity Demand

Annual Demand Projections

Annual provincial electricity demand forecasts are pulled from the CODERS forecasted_annual_demand API endpoint. For each province and period:

• The demand value for the year corresponding to each period is extracted (using data_year(), which returns period + period_step for future periods).
• Values are converted from the CODERS native unit (GWh) to the model unit (PJ) via the conversion factor in units.csv.
• Written to the Demand table.

Hourly Demand Profiles (DemandSpecificDistribution)

For each province with available data for the weather year:

1. Hourly demand data (8,760 values in MWh) is downloaded from CODERS provincial_demand endpoint.
2. A tolerance filter is applied: any hour with demand below dsd_tolerance × mean_demand is set to zero. This is done for computational efficiency in the optimisation solver.
3. The profile is normalised by dividing each hour by the total annual sum, creating a distribution that sums to 1.0 over the year.
4. The normalised profile is written to DemandSpecificDistribution for every model period (the shape is assumed constant across periods; only annual demand changes).

Assumption: The hourly demand shape from the weather year is used identically for all future periods. Only the total annual demand scales with projections. This means structural changes in demand patterns (e.g., electrification shifting load shapes) are not captured.

Assumption: The hourly demand data is stored for use in capacity credit calculations later, linking demand-side data to supply-side reliability metrics.

5.2 Transmission and Distribution

Two dummy technologies represent in-province grid infrastructure:

• Both are given unlimited capacity (unlim_cap = 1) since transmission expansion is not modelled within provinces.
• Variable costs for both transmission and distribution are pulled from the U.S. EIA Annual Energy Outlook (AEO) levelised cost tables. Costs are converted from US cents/kWh (USD 2024) to M$/PJ (CAD 2020) using currency conversion factors.
• Line losses are province-specific, taken from the CODERS ca_system_parameters table (system_line_losses_percent).

Assumption: In-province transmission is not capacity-constrained. The model can move unlimited electricity within a province at the associated variable cost and line loss.

Assumption: Transmission and distribution variable costs are uniform across provinces and taken from U.S. data (EIA AEO), which may not perfectly reflect Canadian cost structures.

5.3 Planning Reserve Margin

Provincial planning reserve margins are taken from CODERS ca_system_parameters (reserve_requirements_percent). These are written directly to the PlanningReserveMargin table for each region.

Assumption: Reserve margins are province-specific but constant over the entire planning horizon.

6. Generator Aggregation

Module: generators.py

This is the most complex module. It aggregates both existing and new generators, filling the bulk of the database tables.

6.1 Existing Generators

Data Source

The CODERS generators API endpoint provides a facility-level database of all Canadian electricity generators, including: - Facility name and code, province, installed capacity, start year, renewal year - Generator type (e.g., "Natural Gas - Combined Cycle"), latitude/longitude - Average annual energy output, capacity factor

Processing Steps

1. Technology mapping: Each CODERS generator type is mapped to a CANOE technology code via config.existing_map (built from generator_technologies.csv's coders_existing column). Generators without a mapping are skipped with a warning.

2. Region mapping: CODERS province names are mapped to CANOE region codes. Only generators in endogenous model regions are included.

3. Capacity filtering: Zero-capacity entries are removed.

4. Vintage determination: The vintage is set to max(start_year, previous_renewal_year). This captures major refurbishments.

5. Vintage rounding: Vintages are rounded to the period_step interval and capped at first_model_period - 1. This reduces the number of unique vintages in the database:

6. Example: with period_step=5 and first period 2025, a generator started in 2007 gets vintage 2005, one from 2013 gets vintage 2015, and one from 2023 gets vintage 2024 (capped at 2024).

7. No-retirement handling: Technologies flagged no_retirement = true (e.g., hydroelectric) have their vintage overridden to first_model_period - 1 and lifetime set to 100 years. This ensures they persist throughout the planning horizon.

8. Aggregation: Capacities are summed by (region, tech_code, vintage). Facility descriptions are concatenated.

9. Filtering:

10. Generators whose vintage + lifetime does not reach the first model period are excluded.

11. Generators below the existing_capacity_threshold (0.001 GW) after aggregation are excluded.

12. Database writing: For each aggregated (region, tech, vintage) combination:

13. Technology table entry with flag, sector, description
14. ExistingCapacity table entry with converted capacity (MW → GW)

15. TimePeriod entries for existing vintages (flag "e")

16. Downstream processing: The aggregated dataframe is passed to capacity factor, capacity credit, constraint, and generic-data modules.

6.2 New Generators

For each technology in generator_technologies.csv with include_new = true:

1. Batching: If new_cap_batches > 1, multiple technology variants are created (e.g., E_WND_ON-NEW-1 through E_WND_ON-NEW-13). Each batch represents a bin of new capacity, typically used for variable renewables with different resource qualities.

2. Technology entries: Written to the Technology table.

3. Region-vintage combinations: A full cross-product of (model_regions × model_periods) is generated as the set of region-tech-vintage entries.

4. Wind and solar are separated from other generators and passed to new_wind_solar.py for specialised provincial resource characterisation. The remaining generators continue through the generic data pipeline.

6.3 Generic Techno-Economic Data

Two parallel paths handle generic data depending on whether an ATB equivalent is defined:

ATB Path (preferred when atb_display_name is set)

Data from the NREL ATB is used for:

• Efficiency: Derived from ATB Heat Rate (1 / (heat_rate × conversion_factor)). For renewable technologies with dummy inputs (ethos), efficiency is set to 1.
• Investment cost (CostInvest): ATB's OCC (Overnight Capital Cost) metric, converted from USD/kW to M$/GW and then to the final currency (CAD 2020). Projected values are taken for each vintage year.
• Fixed O&M (CostFixed): From ATB Fixed O&M, indexed by vintage year. Converted to model units and currency.
• Variable O&M (CostVariable): From ATB Variable O&M. If include_tech_fuel_cost is enabled and the technology has include_fuel_cost = true, fuel costs from ATB are added to the variable cost.
• Ramp rates: From ATB technology-specific variables (TSV) workbook tables if available, otherwise from CODERS.
• Emissions (EmissionActivity): From ATB TSV tables for CO2, SO2, NOx, and Hg, scaled by efficiency. CO2 values are duplicated as CO2-equivalent entries.

CODERS Path (fallback when no ATB equivalent)

Uses CODERS generation_generic and generation_cost_evolution tables for:

• Efficiency, investment cost, fixed O&M, variable O&M, ramp rates, and CO2 emissions.
• All values are converted from CODERS native units (typically CAD/kW, % per minute, etc.) via units.csv conversion factors.

Lifetime

Service lifetime is always taken from the CODERS generation_generic table (service_life column), regardless of ATB/CODERS path, except for no-retirement technologies which get 100 years.

Capacity-to-Activity

A uniform c2a ratio of 31.536 PJ/GW·yr is written for every technology, representing the conversion from nameplate capacity to maximum annual energy output.

Emissions Accounting

Two emissions modes are available:

• include_emissions = false (default): Emissions are assumed to be accounted for upstream (in fuel supply sectors). For technologies with CCS, negative emissions entries are written to offset the captured portion, calculated as -emissions × capture_rate / (1 - capture_rate).
• include_emissions = true: Direct emissions are written as positive values.

Assumption: When emissions are off, the model assumes a complementary fuel-supply sector handles emissions correctly. CCS technologies then only need to represent the incremental capture benefit.

6.4 Technology-Specific Variables (TSV)

The ATB master Excel workbook contains detailed technical parameters in technology-specific variable tables. These are parsed by:

1. Reading the relevant worksheet (e.g., "Natural Gas_FE", "Nuclear") using row/column ranges from atb_master_tables.csv.
2. Concatenating multi-row headers into single-row headers.
3. Translating column names using the tsv_headers mapping in params.yaml.
4. Caching the parsed table locally as a CSV for faster subsequent loads.

7. Capacity Factors

Module: capacity_factors.py (dispatcher), with implementations in provincial_data/

Capacity factors are critical for representing the output variability of generators. The approach varies by technology type and data availability.

7.1 Existing VRE (Wind and Solar) — Ontario

Module: provincial_data/on/existing_vre_capacity_factors.py

For Ontario, real operational data is available from the IESO:

1. Hourly generation by fuel is downloaded from IESO's GenOutputbyFuelHourly XML endpoint for the weather year.
2. The XML data is parsed to extract hourly MWh generation for WIND and SOLAR fuel types.
3. Average annual capacity factors for each technology are computed as capacity-weighted averages from CODERS facility data (facilities ≥20 MW, matching IESO's reporting threshold).
4. Hourly capacity factors are computed by normalising the hourly generation profile to the average annual capacity factor: cf(h) = hourly(h) / mean(hourly) × cf_annual.
5. Values are clipped to [0, 1] and values below cf_tolerance (0.01) are set to zero.
6. Written to CapacityFactorTech for all valid (region, period, season, tod) combinations.

Assumption: Ontario's aggregate VRE generation profile from IESO is representative of the true hourly resource availability of its existing fleet.

7.2 Existing VRE — Other Provinces

Module: provincial_data/default/existing_vre_capacity_factors.py

For provinces without real operational data, capacity factors are synthesised using the Renewables Ninja API:

1. For each existing VRE facility in CODERS, its latitude and longitude are sent to the Renewables Ninja API to obtain simulated hourly generation for the weather year.
2. Results are based on MERRA-2 reanalysis weather data.
3. Facility-specific hourly profiles are aggregated using capacity-weighted averaging.
4. The total annual energy is adjusted to match the CODERS-reported unit_average_annual_energy.
5. Aggregate capacity factors are computed as cf(h) = energy_adjusted_gen(h) / total_capacity.

Assumption: The Renewables Ninja API provides representative profiles using the MERRA-2 weather dataset, a standard reference turbine (Vestas V112 3000 for onshore wind at 110m, Vestas V164 9500 for offshore at 150m), and a 45° tilt for solar PV.

Limitation: API rate limits (50 requests/hour) mean that initial data collection for all facilities can take several hours. Results are cached locally.

7.3 Existing Hydroelectric — Ontario

Module: provincial_data/on/existing_hydro_capacity_factors.py

Ontario hydroelectric capacity factors use IESO's GenOutputCapabilityMonth data:

1. For data before 2019: A single Excel workbook per year provides hourly output and available capacity by generator.
2. For data from 2020 onward: Monthly CSV files provide the same data.
3. Generators are classified as daily storage (hydro_daily) or run-of-river (hydro_run) via a manually curated mapping file (hydro_types.csv).
4. Hourly capacity factors are computed as output_MW / capability_MW for each type.
5. Run-of-river gets hourly CapacityFactorTech entries.
6. Daily storage gets daily-averaged LimitSeasonalCapacityFactor entries (one per season/day).

Assumption: Ontario has no monthly-storage hydroelectric capacity in the IESO dataset. Monthly hydro is handled separately.

7.4 Existing Hydroelectric — Other Provinces

Module: provincial_data/default/existing_hydro_capacity_factors.py

For other provinces, hydroelectric capacity factors are synthesised from Statistics Canada monthly generation data:

1. StatCan table 25-10-0015-01 provides monthly hydraulic turbine generation by province in MWh.
2. Monthly totals are spread uniformly across days within each month.
3. The total provincial hydro generation is apportioned among hydro types (run-of-river, daily storage, monthly storage) based on each type's unit_average_annual_energy from CODERS.
4. Hourly capacity factors are computed from the resulting daily generation divided by aggregated capacity.

Run-of-river hydro is assumed to have constant output within each month (flat hourly profile per month). Daily and monthly storage hydro are given seasonal capacity factor limits (LimitSeasonalCapacityFactor) rather than hourly factors, allowing the optimiser to dispatch them within those bounds.

Assumption: Monthly hydro generation data can be evenly spread over the days of that month. The breakdown between hydro types is based on average annual energy proportions from CODERS.

Limitation: This is a coarse approximation. Real hydro dispatch is driven by water inflows, reservoir levels, and operational decisions — none of which are captured here.

7.5 New Generators

For new non-renewable, non-VRE generators, no capacity factors are set (the optimiser freely dispatches within their other constraints).

For new hydro (daily, run-of-river), Ontario existing hydro capacity factors are currently used as a proxy for all regions. This is a known limitation flagged with a TODO.

8. Capacity Credits and Reserve Margin

Module: capacity_credits.py, provincial_data/on/existing_capacity_credits.py

Capacity credits express how much firm capacity each technology contributes toward meeting the planning reserve margin.

8.1 Non-VRE Generators

Capacity credits for thermal, nuclear, and hydro generators are derived from Ontario's IESO Reliability Outlook:

1. The Reliability Outlook Table 4.1 provides forecast capability at summer peak and total installed capacity by fuel type.
2. The ratio gives a capacity credit: cc = capability / installed_capacity.
3. IESO fuel types are mapped to CANOE generator codes via fuel_types.csv.
4. These credits are written to both CapacityCredit (for static reserve margin) and ReserveCapacityDerate (for dynamic/seasonal reserve margin) tables.

Assumption: Ontario's IESO reliability data is used for all provinces. This is a significant simplification — provinces with different generator fleets may have different de-rating factors.

8.2 VRE Capacity Credits (New Wind and Solar)

VRE capacity credits are calculated using the NREL ReEDS top-100-hours method:

1. Start with the provincial hourly load (demand profile).
2. Subtract hourly generation from all existing VRE to get an initial net load.
3. For each new VRE capacity bin (ordered by ascending LCOE): a. Calculate the marginal net load by subtracting the bin's projected hourly generation (capacity factor × nameplate capacity) from the current net load. b. Compute the load duration curve (LDC) and the net load duration curve (NLDC) (both sorted descending). c. The capacity credit is the mean reduction in NLDC over the top 100 hours, divided by nameplate capacity. d. Update net load for the next iteration.
4. This produces declining marginal capacity credits: the first bins of VRE added have higher capacity value; later bins have diminishing value as their output becomes increasingly correlated with existing VRE.

Assumption: VRE capacity bins are assumed to be built sequentially in order of ascending LCOE, and only one type of renewable is evaluated at a time (cross-technology interactions are not captured).

8.3 Storage Capacity Credits

Storage is currently assigned a flat capacity credit of 0.9 for all regions and periods. This is a placeholder pending a more sophisticated method.

9. New Wind and Solar Characterisation

Module: new_wind_solar.py

New wind and solar use spatially explicit resource characterisation data. This is the most data-intensive part of the pipeline.

9.1 New Wind

Resource Data

Wind resource data comes from characterisation work by Sutubra, providing:

• Cluster Composition: For each geographic cluster, the fraction of three turbine technology classes (T1, T2, T3 corresponding to ATB wind classes 7, 8, 9) and maximum capacity.
• Cluster Capacity Factors: Hourly (8,760) capacity factor profiles per cluster.
• Cluster Spur Costs: Estimated cost to connect each cluster to the existing transmission grid.

Processing Steps

1. ATB cost projections are loaded for each wind class (T1–T3): investment cost (OCC), fixed O&M, and capacity factor projections over time.
2. Weighted cluster costs are computed by weighting ATB data by the capacity fraction of each turbine class in the cluster, then adding spur line costs.
3. LCOE ranking: Clusters are sorted by estimated LCOE using a simple annualised cost / annual energy calculation with the global discount rate.
4. Bin assignment: The top N clusters (N = number of capacity bins from generator_technologies.csv) are assigned to new capacity bins 1–N.
5. Capacity factor indexing: Hourly CFs are scaled by ATB's projected CF improvement relative to a 2030 base year for each vintage.
6. Capacity credits are calculated using the VRE top-100-hours method.
7. Database entries: For each (region, cluster, vintage):
8. LimitCapacity: Maximum capacity per bin (from cluster data, MW → GW)
9. Efficiency: 1.0 (dummy ethos input)
10. CostInvest: Weighted ATB + spur costs, currency-converted
11. CapacityFactorProcess: Hourly CFs for each vintage (indexed to ATB projections)
12. CostFixed: Weighted ATB fixed O&M

Assumption: Currently, only Ontario wind resource data is used for all provinces. This is flagged as a TODO for future improvement.

Assumption: Wind turbine classes 7/8/9 from ATB represent the range of resources available. Cost and performance improvements follow ATB projections.

9.2 New Solar

Similar to wind, but simpler (no turbine class weighting):

1. Solar resource data provides per-grid-cell capacity factors, maximum capacity, interconnection costs, and LCOE.
2. Grid cells are sorted by LCOE (or capacity factor, configurable).
3. The top N cells are assigned to capacity bins.
4. ATB solar PV cost and performance projections are applied, indexed to a 2022 base year.
5. Solar degradation: A per-year degradation rate (default 0.85%/year) is applied to capacity factors, reducing output linearly with age.
6. Capacity credits and database entries are computed as for wind.

Assumption: Solar degradation follows a flat 0.85%/year rate based on Jordan et al. (2016).

Assumption: As with wind, only Ontario solar resource data is currently used for all provinces.

10. Energy Storage

10.1 Existing Storage

From the CODERS storage API endpoint:

1. Each storage facility is mapped to a CANOE technology based on (storage_type, duration).
2. Duration is rounded to the nearest integer hour.
3. Capacities are aggregated by (region, tech, vintage) following the same vintage-rounding and filtering logic as generators.
4. Efficiency is taken from storage_technologies.csv (fixed, ATB-aligned).
5. StorageDuration is set from the facility's reported duration.
6. Technology flag is "ps" (physical storage).

10.2 New Storage

For new storage technologies (with include_new = true):

1. A single technology variant is created per storage type (e.g., E_BAT_2H-NEW, E_BAT_4H-NEW).
2. Efficiency and storage duration come from storage_technologies.csv.
3. Cost data follows the ATB or CODERS path as for generators.

10.3 Monthly Hydroelectric Storage

Monthly hydro is given special treatment to model reservoir storage:

1. An intermediate commodity (E_hyd_mly_stor) is created to represent stored water energy.
2. An inflow technology (E_HYD_MLY-EXS-IN) fills the reservoir — it takes weather-dependent hydro inflow and produces the storage commodity.
3. The existing hydro tech (E_HYD_MLY-EXS) is converted to a seasonal storage technology that takes from the reservoir and produces electricity.
4. StorageDuration is set to 730 hours (approximately one month).
5. The LimitSeasonalCapacityFactor from hydro CF calculations is assigned to the inflow technology rather than the dispatch technology, constraining how much energy enters the reservoir each season while allowing flexible dispatch.

Assumption: 730 hours of storage capacity represents monthly-scale reservoir flexibility. Real reservoirs may have multi-month or multi-year storage.

11. CCS Retrofits

Module: Part of generators.py

CCS retrofits are modelled as add-on technologies that sit between an existing fossil generator and the transmission grid:

Architecture

Generator → [Intermediate Commodity] → CCS Retrofit → Transmission
                                    ↘ Bypass Tech  ↗

1. A new intermediate commodity (e.g., E_elc_tx_ng_cc) is created between the generator output and the transmission grid.
2. A bypass technology allows unabated electricity to pass through at efficiency 1.0.
3. A retrofit technology (e.g., E_NG_CCS_RFIT_95) captures CO2 at a specified capture rate (e.g., 90% or 95%) with an energy penalty.

Data Processing

1. Efficiency: From ATB "Net Output Penalty" metric: efficiency = 1 + penalty (penalty is negative, so efficiency < 1).
2. Emissions: Negative emission entries: -capture_rate × generator_co2_emissions / efficiency.
3. Costs: Investment, fixed O&M, and variable O&M from ATB CCS retrofit data.
4. Lifetime: CCS retrofit lifetime is capped to not exceed the remaining life of the retrofittable generators.
5. Activity limits: Retrofit and bypass activities are capped to zero when no generators remain alive.

Assumption: The ATB provides representative costs for CCS retrofits on natural gas combined cycle and coal plants. Costs are not region-specific.

Assumption: CCS retrofits are vintage-dependent — costs decrease over time following ATB projections.

12. Interfaces and Interties

Module: interfaces.py

Interties (electricity interconnections) between provinces and to the U.S. are aggregated in two categories:

12.1 Boundary Interfaces

These cross the model boundary (one region inside the model, one outside). They are represented as:

• Outgoing demand: A demand commodity and demand technology force the model to export the historically observed flow pattern.
• Incoming VRE generator: A variable generator with curtailment ability represents imports, with capacity factors derived from historical hourly flows.

Processing Steps

1. All interprovincial and international interties are loaded from CODERS.
2. For each intertie crossing the model boundary, hourly flow data is retrieved for the weather year.
3. Flow direction: In CODERS, negative flows represent forward transfers and positive flows represent backward transfers.
4. Flows are aggregated by regional pair.
5. Export (outgoing):
6. Annual demand = sum of outgoing hourly flows (MWh → PJ).
7. Demand profile (DSD) = hourly flow / annual total.
8. Transmission efficiency = 1 − province line loss.
9. Variable cost from AEO transmission costs.
10. Import (incoming):
11. Existing capacity = peak hourly import flow (MWh/h → GW).
12. Capacity factor = hourly flow / peak flow.
13. Marked as curtailable.

Assumption: Historical boundary flows are projected forward as fixed demand/supply patterns for all future periods. This freezes international trade at weather-year levels.

12.2 Endogenous Interfaces

These connect two modelled regions and are represented as exchange technologies in Temoa:

1. A single E_INT technology is used.
2. Capacity: Set to the maximum of seasonal transfer capabilities (summer/winter) in either direction, from CODERS interface_capacities.
3. Seasonal capacity factors: If transfer capabilities differ by season or direction, capacity factors constrain flows below the physical limit.
4. Efficiency: 1 − line loss of the sending province.
5. Variable cost: AEO transmission costs apply.

Assumption: Endogenous interface capacity is fixed at existing levels; no transmission expansion is modelled between provinces.

13. Constraints

Module: constraints.py dispatching to provincial_data/default/

13.1 Ramp Rates

Source: ramp_rates.csv, based on Dolter & Rivers (2018).

Hourly ramp-up and ramp-down rates are applied to technologies based on their CANOE tech code. Values are expressed as a fraction of capacity per hour.

Not all technologies have ramp constraints — only those listed in the CSV file.

13.2 Cogeneration Constraints

Existing cogeneration technologies (natural gas CHP, biomass CHP) are activity-constrained:

• Maximum activity = unit_average_annual_energy from CODERS (converted to PJ).
• Minimum activity = 95% of maximum (computational slack).

This ensures cogeneration output matches historical levels, since the model does not endogenously represent the heat demand served by CHP.

Assumption: Cogeneration heat demand is exogenous and constant. The 5% slack prevents infeasibility.

14. Currency Conversion

Module: currency_conversion.py

All costs in the database are converted to a single target currency and year (default: CAD 2020).

Methodology

The conversion uses two tables from input_files/:

• currency_exchange.csv: Exchange rates for multiple currencies (USD, EUR, GBP, AUD, CAD) by year.
• cad_inflation.csv: Inflation indices (GDP deflator, CPI variants, construction indices) by year.

The conversion formula is:

converted_cost = original_cost × exchange_rate(orig_year, orig_currency) × inflation_index(orig_year) / base_factor

where base_factor = exchange_rate(base_year, base_currency) × inflation_index(base_year).

The GDP deflator index is used by default. Conversion happens inline during data aggregation — each cost value is converted before being written to the database.

15. Post-Processing

Module: post_processing.py

After all data has been aggregated:

15.1 Import Dummy Technologies

If include_imports is enabled, dummy import technologies are created for each fuel commodity used in the model:

1. For every commodity that appears as an input in the Efficiency table, an import technology is created.
2. Import techs have efficiency 1.0, a single vintage at the first period, and retire when no longer needed.
3. This bridges the gap between fuel supply sectors and the electricity sector.

15.2 Commodity Table

Commodities are added to the Commodity table based on actual usage:

• Only commodities appearing in Efficiency (as inputs or outputs) or EmissionActivity (as emission commodities) are added.
• This prevents orphaned commodities in the database.

15.3 Unused Technology Removal

Technologies that appear in the Technology table but never in the Efficiency table are removed from all database tables. This cleans up technologies that were configured but never had valid data (e.g., a province might not have solar generators to aggregate).

15.4 References and Data IDs

• All bibliography entries are written to the DataSource table.
• All unique data IDs are written to the DataSet table.
• A consistency check warns about any database rows with NULL data_id values.

16. Unit Conventions

All unit conversions are centralised in input_files/units.csv. Key conversions include:

The units.csv file contains separate columns for CODERS and ATB conversion factors, since the two data sources use different native units.

17. Known Assumptions and Limitations

Spatial

1. Ontario data as default: Several province-specific calculations (hydro CFs, VRE resource characterisation, capacity credits) currently use Ontario data for all provinces. This is flagged in the code with TODO comments.
2. Province-level granularity: Generators, demand, and transmission are modelled at the provincial level. Sub-provincial variation is not captured.
3. U.S. trade frozen: Boundary interties with the U.S. use weather-year historical flows for all future periods.

Temporal

1. Weather year: All hourly profiles (demand, generation, interties) come from a single weather year (2018 by default). Climate variability is not represented.
2. Static demand shape: The hourly demand profile is assumed constant across all future periods.

Technical

1. No transmission expansion: In-province transmission and interprovincial capacities are fixed.
2. Hydro approximation: Monthly hydro uses 730-hour storage duration; real reservoir operations are more complex.
3. Cogeneration heat demand: Treated as exogenous and constant.
4. Storage capacity credits: Flat 0.9 for all storage, pending improved methodology.

Economic

1. U.S. cost data: Transmission/distribution costs come from EIA AEO, not Canadian sources.
2. ATB prevalence: Technology costs primarily follow U.S. ATB projections, which may not reflect Canadian market conditions.
3. Currency conversion: Uses GDP deflator for inflation adjustment; other indices are available but not default.

Data Quality

1. CODERS data completeness: Some generators may lack renewal dates, capacity factors, or average annual energy data, leading to potential inaccuracies in vintage assignment or CF calculations.
2. Renewables Ninja synthesis: Non-Ontario VRE CFs are simulated rather than observed, introducing weather model uncertainty.

This documentation describes the CANOE electricity sector aggregation tool as of 2025. The tool is under active development and some features (marked with TODO) are planned for future improvement.

Hendriks, R.M., Monroe, J., Cusi, T., Aldana, D., Griffiths, K., Dorman, T., Chhina, A., 
McPherson, M. (2023). Canadian Open-Source Database for Energy Research and Systems-Modelling 
(CODERS). Available at www.cme-emh/coders. {table} table. Accessed {date}

IESO. ({year}). IESO public data. http://reports.ieso.ca/public/
IESO. ({year}, {month}). Reliability Outlook. https://www.ieso.ca/en/Sector-Participants/Planning-and-Forecasting/Reliability-Outlook

Government of Canada, Statistics Canada. Electric power generation, monthly generation by type of 
electricity. Table 25-10-0015-01. https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=2510001501

Pfenninger, S., & Staffell, I. (2016). Long-term patterns of European PV output using 30 years 
of validated hourly reanalysis and satellite data. Energy, 114, 1251–1265. 
https://doi.org/10.1016/j.energy.2016.08.060

U.S. Energy Information Administration. Annual Energy Outlook {year}. 
https://www.eia.gov/outlooks/aeo/data/browser/

Dolter, B., & Rivers, N. (2018). The cost of decarbonizing the Canadian electricity system. 
Energy Policy, 113, 135–148. https://doi.org/10.1016/j.enpol.2017.10.040

Jordan, D. C., Kurtz, S. R., VanSant, K., & Newmiller, J. (2016). Compendium of photovoltaic 
degradation rates. Progress in Photovoltaics, 24(7), 978–989. https://doi.org/10.1002/pip.2744

Frew, B., Cole, W., Sun, Y., Richards, J., & Mai, T. (2017). 8760-Based Method for Representing 
Variable Generation Capacity Value in Capacity Expansion Models. NREL. 
https://www.nrel.gov/docs/fy17osti/68869.pdf