Last updated: 2021-01-24
Checks: 7 0
Knit directory: MHWflux/
This reproducible R Markdown analysis was created with workflowr (version 1.6.2). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.
Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.
Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.
The command set.seed(666) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.
Great job! Recording the operating system, R version, and package versions is critical for reproducibility.
Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.
Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.
Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.
The results in this page were generated with repository version 6554bc8. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.
Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:
Ignored files:
Ignored: .Rhistory
Ignored: .Rproj.user/
Ignored: data/ALL_anom.Rda
Ignored: data/ALL_other.Rda
Ignored: data/ALL_ts_anom.Rda
Ignored: data/ERA5_down.Rda
Ignored: data/ERA5_down_anom.Rda
Ignored: data/ERA5_evp_anom.Rda
Ignored: data/ERA5_lhf_anom.Rda
Ignored: data/ERA5_lwr_anom.Rda
Ignored: data/ERA5_mslp_anom.Rda
Ignored: data/ERA5_pcp_anom.Rda
Ignored: data/ERA5_qnet_anom.Rda
Ignored: data/ERA5_shf_anom.Rda
Ignored: data/ERA5_swr_MLD.Rda
Ignored: data/ERA5_swr_anom.Rda
Ignored: data/ERA5_t2m_anom.Rda
Ignored: data/ERA5_tcc_anom.Rda
Ignored: data/ERA5_u_anom.Rda
Ignored: data/ERA5_v_anom.Rda
Ignored: data/GLORYS_all_anom.Rda
Ignored: data/OISST_all_anom.Rda
Ignored: data/packet.Rda
Ignored: data/som.Rda
Ignored: data/synoptic_states.Rda
Ignored: data/synoptic_states_other.Rda
Untracked files:
Untracked: data/event_overlap_res.Rda
Unstaged changes:
Modified: output/magnitude_schematic.pdf
Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.
These are the previous versions of the repository in which changes were made to the R Markdown (analysis/som.Rmd) and HTML (docs/som.html) files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view the files as they were in that past version.
| File | Version | Author | Date | Message |
|---|---|---|---|---|
| Rmd | 6554bc8 | robwschlegel | 2021-01-24 | Re-built site. |
| Rmd | cd0c6fb | robwschlegel | 2021-01-23 | A near complete answer on SOM overlapping. |
| Rmd | b1b4b15 | robwschlegel | 2021-01-23 | Commit before restructuring overlap pipeline |
| Rmd | 2b0b38c | robwschlegel | 2021-01-23 | Working on finding overlapping MHWs |
| html | 57e2ce3 | robwschlegel | 2021-01-23 | Build site. |
| Rmd | 3f6b550 | robwschlegel | 2021-01-23 | Re-built site. |
| Rmd | 0d10736 | robwschlegel | 2021-01-23 | Preparing text to explain the needed additional analyses from reviewers |
| html | 76b55b6 | robwschlegel | 2020-12-21 | Build site. |
| html | 4a00400 | robwschlegel | 2020-12-21 | Build site. |
| html | 65f38bf | robwschlegel | 2020-12-21 | Build site. |
| html | 33f4595 | robwschlegel | 2020-11-10 | Build site. |
| html | a438235 | robwschlegel | 2020-11-10 | Build site. |
| Rmd | cbc5b74 | robwschlegel | 2020-11-10 | Re-built site. |
| Rmd | e513e01 | robwschlegel | 2020-10-09 | Added Qsw* to the SOM data and removed the SOM correlation figure |
| Rmd | 54144a7 | robwschlegel | 2020-09-28 | Working on non-numeric node labels |
| Rmd | 5bfa2a7 | robwschlegel | 2020-09-03 | Push before beginning large structural additions to app |
| html | 8d65758 | robwschlegel | 2020-09-03 | Build site. |
| Rmd | d1c9bad | robwschlegel | 2020-09-03 | Re-built site. |
| Rmd | 66f3736 | robwschlegel | 2020-08-26 | More edits to the figures |
| Rmd | 4b04d7a | robwschlegel | 2020-08-14 | Renamed some files in preparation for the file runs on the SOM sized data |
| Rmd | c0c599d | robwschlegel | 2020-08-12 | Combining the MHWNWA and MHWflux code bases |
This vignette contains the code used to perform the self-organising map (SOM) analysis on the mean synoptic states created in the data preparation vignette. We’ll start by creating custom packets that meet certain experimental criteria before feeding them into a SOM.
# Load functions and objects to be used below
source("code/functions.R")In this step we will create a data packet that can be fed directly into the SOM algorithm. This means that it must be converted into a super-wide matrix format. In the first run of this analysis on the NAPA model data it was found that the inclusion of the Labrador Sea complicated the results quite a bit. It was also unclear whether or not the Gulf of St Lawrence (GSL) region should be included in the analysis. So in the second run of this analysis multiple different SOM variations were employed and it was decided that the GSL region should be included.
Up first we must create the synoptic state packets.
# Set number of cores
# NB: 50 cores uses too much RAM
registerDoParallel(cores = 20)
# Load needed data
ALL_anom <- readRDS("data/ALL_anom.Rda")
ALL_other <- readRDS("data/ALL_other.Rda")
# Create one big anomaly packet from OISST data
system.time(synoptic_states <- plyr::ddply(OISST_MHW_event, c("region", "event_no"),
data_packet_func, .parallel = T, df = ALL_anom)) # 129 seconds
# Save
saveRDS(synoptic_states, "data/synoptic_states.Rda")
# Create other synoptic states per MHW per variable
doParallel::registerDoParallel(cores = 10) # NB: Be careful here...
system.time(synoptic_states_other <- plyr::ddply(OISST_MHW_event, c("region", "event_no"),
data_packet_func, .parallel = T, df = ALL_other)) # 212 seconds
# Save
saveRDS(synoptic_states_other, "data/synoptic_states_other.Rda")With all of our data ready we may now prepare and save them for the SOM.
## Create wide data packet that is fed to SOM
system.time(packet <- synoptic_states %>%
select(region, event_no, synoptic) %>%
unnest(cols = "synoptic") %>%
wide_packet_func()) # 79 seconds
# Save
saveRDS(packet, "data/packet.Rda")Now we feed the SOM with a function that ingests the data packet and produces results for us. The function below has been greatly expanded on from the previous version of this project and now performs all of the SOM related work in one go. This allowed me to remove a couple hundreds lines of code and text from this vignette.
# # OISST SOM analysis
packet <- readRDS("data/packet.Rda")
synoptic_states_other <- readRDS("data/synoptic_states_other.Rda")
system.time(som <- som_model_PCI(packet, synoptic_states_other)) # 176 seconds
saveRDS(som, file = "data/som.Rda")
# saveRDS(som, file = "shiny/som.Rda")A reviewer of the manuscript noted that the MHWs appear to be clustered closely together both within and across regions. This is by design in the methodology, but just how exactly these events cluster together in the SOM nodes warrants further investigation. The answer of how closely events cluster together within each region was answered in the MHWs vs. heat flux vignette. In this section we will first create an index of MHWs that can be said to be occurring across multiple regions at once. We then look to see how often these events are clustered into the same, or different nodes.
# Double for loop to crawl through events and find overlapping events
# So hideous...
event_overlap_res <- data.frame()
for(i in 1:nrow(OISST_MHW_event)){
df1 <- OISST_MHW_event[i,]
df1_dates <- seq(df1$date_start, df1$date_end, by = "day")
OISST_MHW_event_sub <- filter(OISST_MHW_event, region != df1$region)
for(j in 1:nrow(OISST_MHW_event_sub)){
df2 <- OISST_MHW_event_sub[j,]
df2_dates <- seq(df2$date_start, df2$date_end, by = "day")
if(any(df2_dates %in% df1_dates)){
# Prep some info
longest_dur <- max(c(df1$duration, df2$duration))
overlap <- sum(df2_dates %in% df1_dates)
overlap_prop <- overlap/longest_dur
# Create the full info
event_overlap <- df2 %>%
dplyr::rename(region_match = region,
event_no_match = event_no,
duration_match = duration) %>%
mutate(region = df1$region,
event_no = df1$event_no,
duration = df1$duration,
date_peak_orig = df1$date_peak) %>%
dplyr::select(region, event_no, duration, region_match,
event_no_match, duration_match:date_end, date_peak_orig) %>%
mutate(overlap = overlap,
overlap_prop = overlap_prop,
strng1 = paste(region, event_no, region_match, event_no_match),
strng2 = paste(region_match, event_no_match, region, event_no))
# Check that the pairing hasn't already been counted
if(!event_overlap$strng2 %in% event_overlap_res$strng1){
event_overlap_res <- rbind(event_overlap_res, event_overlap)
}
}
}
}; rm(i, j)# The events per node
SOM <- readRDS("data/som.Rda")
SOM_info <- SOM$info %>%
mutate(node = LETTERS[node])
# How well do these overlaps match up?
event_overlap_node <- event_overlap_res %>%
left_join(SOM_info[,c("region", "event_no", "node")],
by = c("region", "event_no")) %>%
left_join(SOM_info[,c("region", "event_no", "node")],
by = c("region_match" = "region", "event_no_match" = "event_no")) %>%
mutate(match = node.x == node.y)
# Get the total count of overlapping events clustered into the same node
paste0(sum(event_overlap_node$match), "/",
nrow(event_overlap_node))[1] "190/321"# The number of matched events not clustered together when one event is 10 days or shorter
nrow(filter(event_overlap_node, match == FALSE, duration <= 10 | duration_match <= 10))[1] 74# The mean and median proportion of overlap of evnts not clustered together
event_overlap_node %>%
ungroup() %>%
filter(match == FALSE) %>%
summarise(mean_overlap_prop = mean(overlap_prop),
median_overlap_prop = median(overlap_prop))# A tibble: 1 x 2
mean_overlap_prop median_overlap_prop
<dbl> <dbl>
1 0.262 0.214# The mean and median proportion of overlap of clustered events
event_overlap_node %>%
ungroup() %>%
filter(match == TRUE) %>%
summarise(mean_overlap_prop = mean(overlap_prop),
median_overlap_prop = median(overlap_prop))# A tibble: 1 x 2
mean_overlap_prop median_overlap_prop
<dbl> <dbl>
1 0.503 0.468This result shows us that of the 321 overlapping event pairs, 190 of them are clustered into the same nodes as one another. Looking through the results manually one may see that many of the overlapping events that are not clustered together, one of the events is short and the other is longer. Indeed, of the 131 overlapping events that are not clustered together, 74 of these pairs have at least one of the events being 10 days or shorter in duration. Additionally, the median proportion of overlapping days for events not clustered together is 0.21, but 0.47 for clustered events. We can conclude from this that when synoptic scale patterns are causing multiple MHWs simultaneously across regions that the SOM clusters these correctly, with the exception of when one of the events is very short, or the events don’t overlap by very many days. For the shorter events this is likely because the cause of these events is a more transient driver that isn’t resolved as clearly in the mean synoptic state for the longer event. When the events don’t overlap by much, they will generally be related to different drivers that are emerging in close temporal proximity, but are not the same phenomenon.
Another minor point raised by a reviewer was that the atmospheric pattern in Node I appears to precede that of Node L, and they wanted to know if the MHWs clustered into Node I actually were coming before those events in Node L, and that these two nodes were really just a continuation of a single atmospheric pattern. To determine this we find the gaps in occurrence of the start, peak, and end dates of the MHWs between these nodes. First, we can simply look at the results from the above chunk and confirm that there are indeed no events in Node I that overlap in time with Node L.
# Prep data
OISST_MHW_event_node <- left_join(OISST_MHW_event,
SOM_info[,c("region", "event_no", "season_peak", "node")],
by = c("region", "event_no", "season" = "season_peak")) %>%
dplyr::select(region, event_no, node, duration, date_start:date_end)
events_I <- filter(OISST_MHW_event_node, node == "I") %>%
left_join(event_overlap_node)
events_L <- filter(OISST_MHW_event_node, node == "L")
# Find the nearest occurring events between nodes
dist_days <- function(df){
# Convert dates to integers for FNN
df_int <- df %>%
mutate(date_start = as.integer(date_start),
date_peak = as.integer(date_peak),
date_end = as.integer(date_end))
events_L_int <- events_L %>%
mutate(date_start = as.integer(date_start),
date_peak = as.integer(date_peak),
date_end = as.integer(date_end))
# Find nearest event
res <- events_L[as.numeric(knnx.index(as.matrix(events_L_int[,c("date_start", "date_peak", "date_end")]),
as.matrix(df_int[,c("date_start", "date_peak", "date_end")]), k = 1)),] %>%
mutate(region_I = df$region,
event_no_I = df$event_no,
duration_I = df$duration,
date_start_I = df$date_start,
date_peak_I = df$date_peak,
date_end_I = df$date_end,
start_diff = date_start - date_end_I)
return(res)
}
event_day_dist <- plyr::ddply(events_I, c("region", "event_no"), dist_days)
# The gaps between the nearest events between the nodes
event_day_dist$start_diff[order(event_day_dist$start_diff)]Time differences in days
[1] -676 -674 -643 -632 -627 -19 -18 3 25 117 638 663 666 2159 2513
[16] 2771 3110 3503From these results we see that the events occurring in Node I and L usually occurred very far apart from one another. Nearly two years or more in most cases. Remember that the average time distance between events in the MHWs for this study is ~ two months, so this is a very large difference relative to the average. We do however see that there are four values falling within one month of each other. Looking at these values more closely one sees that these are four events from Node I relating to two events in Node L as the Node I events are in two sets of two. Surprisingly, one of these near matches shows that the Node L event occurred before the two nearby Node I events. This implies that the massive low pressure system in Node L was followed by the low pressure sytem in Node I, and that these were two separate patterns.
sessionInfo()R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
locale:
[1] LC_CTYPE=en_CA.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_CA.UTF-8 LC_COLLATE=en_CA.UTF-8
[5] LC_MONETARY=en_CA.UTF-8 LC_MESSAGES=en_CA.UTF-8
[7] LC_PAPER=en_CA.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] doParallel_1.0.16 iterators_1.0.13 foreach_1.5.1 FNN_1.1.3
[5] Metrics_0.1.4 yasomi_0.3 proxy_0.4-24 e1071_1.7-4
[9] ggraph_2.0.4 correlation_0.5.0 tidync_0.2.4 heatwaveR_0.4.5
[13] lubridate_1.7.9.2 data.table_1.13.6 forcats_0.5.0 stringr_1.4.0
[17] dplyr_1.0.2 purrr_0.3.4 readr_1.4.0 tidyr_1.1.2
[21] tibble_3.0.4 ggplot2_3.3.3 tidyverse_1.3.0
loaded via a namespace (and not attached):
[1] fs_1.5.0 insight_0.12.0 httr_1.4.2 rprojroot_2.0.2
[5] tools_4.0.3 backports_1.2.1 utf8_1.1.4 R6_2.5.0
[9] DBI_1.1.1 colorspace_2.0-0 withr_2.3.0 tidyselect_1.1.0
[13] gridExtra_2.3 compiler_4.0.3 git2r_0.28.0 cli_2.2.0
[17] rvest_0.3.6 RNetCDF_2.4-2 xml2_1.3.2 bayestestR_0.8.0
[21] scales_1.1.1 digest_0.6.27 rmarkdown_2.6 pkgconfig_2.0.3
[25] htmltools_0.5.1 dbplyr_2.0.0 rlang_0.4.10 readxl_1.3.1
[29] rstudioapi_0.13 farver_2.0.3 generics_0.1.0 jsonlite_1.7.2
[33] magrittr_2.0.1 ncmeta_0.3.0 parameters_0.10.1 Rcpp_1.0.5
[37] munsell_0.5.0 fansi_0.4.1 viridis_0.5.1 lifecycle_0.2.0
[41] stringi_1.5.3 whisker_0.4 yaml_2.2.1 MASS_7.3-53
[45] plyr_1.8.6 grid_4.0.3 promises_1.1.1 ggrepel_0.9.0
[49] crayon_1.3.4 graphlayouts_0.7.1 haven_2.3.1 hms_1.0.0
[53] knitr_1.30 pillar_1.4.7 igraph_1.2.6 effectsize_0.4.1
[57] codetools_0.2-18 reprex_0.3.0 glue_1.4.2 evaluate_0.14
[61] modelr_0.1.8 vctrs_0.3.6 tweenr_1.0.1 httpuv_1.5.5
[65] cellranger_1.1.0 gtable_0.3.0 polyclip_1.10-0 assertthat_0.2.1
[69] xfun_0.20 ggforce_0.3.2 broom_0.7.3 tidygraph_1.2.0
[73] later_1.1.0.1 class_7.3-17 ncdf4_1.17 viridisLite_0.3.0
[77] workflowr_1.6.2 ellipsis_0.3.1