Optional outputs
The diem function can output additional files useful for
tracking algorithm convergence during iterations and ensuring repeatable
initialisation of genome polarisation. The flowchart in Figure 1 specifies what processes generate the files
and how the user controls what files will be stored and where.
Figure 1. Flowchart of how to control output files and
their location in diem. Green – functions generating or
using the files. Except diem all functions are internal.
Beige, grey – variable values set by the user (grey) or by internal
processes (beige). Orange – stored output files. Yellow rectangles –
main processes generating files.
The user perspective
The verbose argument in the diem function
controls the stored output files. The default value
verbose = FALSE writes three key output
files in the current working directory (Figure
1). To locate the files, the current working directory can be found
with:
Additionally, the verbose argument can trigger a verbose
output in two ways.
- When
verbosecontains a character string with a path to a specific folder, the verbose output will be stored in that location. - When
verbose = TRUE, the verbose output will be stored in the current working directory.
NullMarkerPolarities
The optimal approach to genome polarisation with diem is
to strip all possible data cues before the analysis. In this way, if
diem finds a barrier you can be certain it is an intrinsic
property of the data. Using random initial marker polarities is central
to this approach. We recommend setting the diem argument
markerPolarity = FALSE. The function then generates random
polarities for the initiation of the algorithm iterations.
This initial state for diem is saved in file
NullMarkerPolarities.txt. It is a text file the first line
describing the contents of the file. The following lines show
space-separated TRUE or FALSE values, where each row corresponds to a
compartment as they were specified in the files argument in
diem. Because diem is a deterministic
algorithm, if you start it from the same initial state, you will get
precisely the same answer. You can test this by setting
markerPolarity to be the list from
NullMarkerPolarities.
nullPolarities <- readLines("folder/NullMarkerPolarities.txt")[-1]
nullPolarities <- lapply(
strsplit(nullPolarities, split = " "),
as.logical
)
diem(..., markerPolarities = nullPolarities)You can also test the convergence of the algorithm from different
initial states by running the same analysis, but each time generating a
new random null by setting markerPolarities = FALSE.
MarkersWithChangedPolarities
The file MarkersWithChangedPolarities.txt is a tab-delimited file with three columns and a variable number of rows. The column changedMarkers shows indices of markers across all compartments for which the polarity was changed in the given iteration compared to the marker polarity in the previous iteration. The column time gives a time stamp of when the changes were recorded. The column iteration specifies the iteration of the algorithm.
Note that here, markers are pooled across all compartments. The information is useful in tracking convergence in postprocessing, and so is not of interest to the casual user. During the analysis run, keeping the information about polarity changes separate for each compartment is computationally more tractable, especially in parallel processing. A set of files specifying the information for each compartment is stored in a likelihood folder (Figure 1).
MarkersWithChangedPolarities-(compartmentNumber)
The contents of the files is analogical as in the MarkersWithChangedPolarities.txt. The difference being that the marker indices are sequential for each compartment.
MarkerDiagnostics(iteration)-(compartmentNumber)
The contents of the files correspond to the contents of the MarkerDiagnosticsWithOptimalPolarities.txt. Here, the polarities, diagnostic indices and their supports are reported separately for each iteration and for each compartment.
Both sets of compartment-specific temporary files are initially
stored regardless of the value in the verbose argument, as
they are used internally during the analysis. When
verbose = FALSE, diem attempts to delete the
likelihood folder on exit.
Iteration-specific graphics
The internal ModelOfDiagnostic function implements Eq.
19 (Baird et al. 2023) that is the
powerful tool to detect the most prominent barrier to geneflow in the
data. Seeing how individuals become separated on either side of the
barrier provides insights into the algorithm convergence.
WarpSwitch files
The WarpSwitch.pdf files plot differences in sorted rescaled hybrid indices (Eq. 12) used to identify the barrier to geneflow (Baird et al. 2023). The WarpSwitch.txt files then describe what genomic states the ideal diagnostic marker would have for all included individuals (Eq. 13), and how the ideal 4-genomic state count matrix (Eq. 14) would look like for the first ten individuals.
Hybrid index files
The SortedRescaledHybridIndex.pdf shows how the differences in hybrid indices between individuals belonging to either side of the barrier become more pronounced with progressing iterations. Note that in order to identify which individuals are placed where, one needs to replicate the figure from hybrid index values in files HI.txt.
The files RescaledHItobetaOfRescaledHI.pdf show the weight
for the hybrid index (Eq. 15) that together with the parameter value in
the diem argument epsilon will inform on how
much emphasis the analysis puts on the ideal state with the center of
the barrier identified in the WarpSwich files.
The visualisation in the iteration-specific graphics provide a tangible track record of how the theoretical ideals of pure individuals and perfectly diagnostic markers contribute to informing on a barrier to geneflow in a realistic scenario, where there are no pure individuals and no perfectly diagnostic markers.
The log file
The log.txt file provides a human-readable overview of the
diem iterations. During initialisation stage, the log file
will list the number of markers and individuals in the analysis run,
followed by the top seven rows of the 4-genomic state
count matrix \({}^4\mathbf{I}\)
with null polarisation, correction for small data (Eq. 5), top seven
rows of the \({}^4\mathbf{A}\), which
is the \({}^4\mathbf{I}\) matrix
corrected for compartment- and individual-specific ploidies and is used
to calculate the hybrid index, the top seven rows of the weighted \({}^4\mathbf{V}\) matrix (Eq. 16), and the
respective log-likelihood values (Eq. 19).
For subsequent iterations, the log file also quantifies how many markers switch polarities between iterations, and shows the corresponding change to the \({}^4\mathbf{I}\).