This vignette introduces some of the development features of the isoreader package and is aimed primarily at code contributors interested in expanding its functionality or helping with bug fixes.

Adding new file format readers

Testing out new file format readers is easiest by registering a new reader function for a specific file extension using iso_register_dual_inlet_file_reader and iso_register_continuous_flow_file_reader, respectively. Both require an extension (e.g. ".ext"), name of the new reader function ("new_reader"), and optionally a description. Both functions automatically return a data frame with a list of all registered reader. Overwriting of existing readers with a different function requires an explicit overwrite = TRUE flag. All reader functions must accept an isoreader data structure object (ds) as the first argument, a list of reader specific options as the second argument (options), and should return the structure with data filled in for downstream isoreader operations to work smoothly. The following minimal example illustrates how to do this with the new_reader function simply printing out the layout of the provided data structure skeleton ds.

new_reader <- function(ds, options = list()) {
  isoreader:::log_message("this is the new reader!")
  str(ds)
  return(ds)
}

# register new reader
readers <- iso_register_dual_inlet_file_reader(".new.did", "new_reader")
knitr::kable(readers)
type call extension func cacheable post_read_check description software env
dual inlet iso_read_dual_inlet .caf iso_read_caf TRUE TRUE Dual Inlet file format (older) Isodat isoreader
dual inlet iso_read_dual_inlet .did iso_read_did TRUE TRUE Dual Inlet file format (newer) Isodat isoreader
dual inlet iso_read_dual_inlet .txt iso_read_nu TRUE TRUE Dual Inlet file format Nu isoreader
continuous flow iso_read_continuous_flow .cf iso_read_cf TRUE TRUE Continuous Flow file format (older) Isodat isoreader
continuous flow iso_read_continuous_flow .dxf iso_read_dxf TRUE TRUE Continuous Flow file format (newer) Isodat isoreader
continuous flow iso_read_continuous_flow .iarc iso_read_flow_iarc TRUE TRUE Continuous Flow data archive ionOS isoreader
scan iso_read_scan .scn iso_read_scn TRUE TRUE Scan file format Isodat isoreader
continuous flow iso_read_continuous_flow .cf.rds iso_read_rds FALSE FALSE R Data Storage isoreader isoreader
dual inlet iso_read_dual_inlet .di.rds iso_read_rds FALSE FALSE R Data Storage isoreader isoreader
scan iso_read_scan .scan.rds iso_read_rds FALSE FALSE R Data Storage isoreader isoreader
dual inlet iso_read_dual_inlet .new.did new_reader TRUE TRUE NA NA R_GlobalEnv

# copy an example file from the package with the new extension
iso_get_reader_example("dual_inlet_example.did") %>% file.copy(to = "example.new.did")
#> [1] TRUE

# read the file
iso_read_dual_inlet("example.new.did", read_cache = FALSE)
#> Info: preparing to read 1 data files (all will be cached)...
#> Info: reading file 'example.new.did' with '.new.did' reader...
#> Info: this is the new reader!
#> List of 7
#>  $ version          :Classes 'package_version', 'numeric_version'  hidden list of 1
#>   ..$ : int [1:3] 1 3 1
#>  $ read_options     :List of 4
#>   ..$ file_info        : logi TRUE
#>   ..$ method_info      : logi TRUE
#>   ..$ raw_data         : logi TRUE
#>   ..$ vendor_data_table: logi TRUE
#>  $ file_info        : tibble [1 × 6] (S3: tbl_df/tbl/data.frame)
#>   ..$ file_id      : chr "example.new.did"
#>   ..$ file_root    : chr "."
#>   ..$ file_path    : chr "example.new.did"
#>   ..$ file_subpath : chr NA
#>   ..$ file_datetime: POSIXct[1:1], format: NA
#>   ..$ file_size    : int 134446
#>  $ method_info      : list()
#>  $ raw_data         : tibble [0 × 0] (S3: tbl_df/tbl/data.frame)
#>  Named list()
#>  $ vendor_data_table: tibble [0 × 0] (S3: tbl_df/tbl/data.frame)
#>  Named list()
#>  $ bgrd_data        : tibble [0 × 0] (S3: tbl_df/tbl/data.frame)
#>  Named list()
#>  - attr(*, "class")= chr [1:2] "dual_inlet" "iso_file"
#>  - attr(*, "problems")= tibble [0 × 3] (S3: tbl_df/tbl/data.frame)
#>   ..$ type   : chr(0) 
#>   ..$ func   : chr(0) 
#>   ..$ details: chr(0)
#> Info: finished reading 1 files in 0.32 secs
#> Dual inlet iso file 'example.new.did': 0 cycles, 0 ions ()
file.remove("example.new.did")
#> [1] TRUE

Note that for parallel processing to work during the read process (parallel = TRUE), isoreader needs to know where to find the new reader function. It will figure this out automatically as long as the function name is unique but if this fails (or to be on the safe side), please specify e.g. env = "R_GlobalEnv" or env = "newpackage" during the reader registration. Also note that isoreader will not automatically know where to find all functions called from within the new reader function if they are not part of base R and it is recommended to make all outside calls explicit (e.g. dplyr::filter(...)) to preempt this potential problem. For info messages and warnings to work with the progress bar and in parallel reads, make sure to use isoreader:::log_message(...) and isoreader:::log_warning(...) instead of base R’s message(...) and warning(...).

If you have designed and tested a new reader, please consider contributing it to the isoreader github repository via pull request.

Processing hooks

Isoreader defines two processing hooks at the beginning and end of reading an individual file. This is useful for integration into pipelines that require additional output (such as GUIs) but is also sometimes useful for debugging purposes. The expressions are evaluated in the context of the isoreader:::read_iso_file function and have access to all parameters passed to this function, such as e.g. file_n and path. Same as for new readers: for info messages and warnings to work with the progress bar and in parallel reads, make sure to use isoreader:::log_message(...) and isoreader:::log_warning(...) instead of base R’s message(...) and warning(...). The main difference between the two is that log_message() will honor the quiet = TRUE flag passed to the main iso_read...() call whereas log_warning() will always show its message no matter the quiet setting.

isoreader:::set_read_file_event_expr({
  isoreader:::log_message(sprintf("starting file #%.d, named '%s'", file_n, basename(path)))
})
isoreader:::set_finish_file_event_expr({
  isoreader:::log_message(sprintf("finished file #%.d", file_n))
})

c(
  iso_get_reader_example("dual_inlet_example.did"),
  iso_get_reader_example("dual_inlet_example.caf")
) %>% iso_read_dual_inlet(read_cache = FALSE)
#> Info: preparing to read 2 data files (all will be cached)...
#> Info: reading file 'dual_inlet_example.did' with '.did' reader...
#> Info: starting file #1, named 'dual_inlet_example.did'
#> Info: finished file #1
#> Info: reading file 'dual_inlet_example.caf' with '.caf' reader...
#> Info: starting file #2, named 'dual_inlet_example.caf'
#> Info: finished file #2
#> Info: finished reading 2 files in 4.10 secs
#> Data from 2 dual inlet iso files: 
#> # A tibble: 2 × 6
#>   file_id    raw_data       file_info  method_info  vendor_data_tab… file_path  
#>   <chr>      <glue>         <chr>      <chr>        <chr>            <chr>      
#> 1 dual_inle… 7 cycles, 6 i… 16 entries standards, … 7 rows, 8 colum… dual_inlet…
#> 2 dual_inle… 8 cycles, 6 i… 22 entries standards, … 8 rows, 9 colum… dual_inlet…

isoreader:::initialize_options() # reset all isoreader options

Debugging isoreader

The best way to start debugging an isoreader call is to switch the package into debug mode. This is done using the internal iso_turn_debug_on() function. This enables debug messages, turns caching off by default so files are always read anew, and makes the package keep more information in the isofile objects. It continues to catch errors inside file readers (keeping track of them in the problems) unless you set iso_turn_debug_on(catch_errors = FALSE), in which case no errors are caught and stop the processing so you get the full traceback and debugging options of your IDE.

Debugging binary file reads (Isodat)

Errors during the binary file reads usually indicate the approximate position in the file where the error was encountered. The easiest way to get started on figuring out what the file looks like at that position is to use a binary file editor and jump to the position. For a sense of the interpreted structure around that position, one can use the internal function map_binary_structure which tries to apply all frequently occurring binary patterns recognized by isoreader. The binary representation of the source file is only available if in debug mode but if debug mode is ON, it can be accessed as follows:

# turn on debug mode
isoreader:::iso_turn_debug_on()
#> Info: debug mode turned on, error catching turned on, caching turned off
# read example file
ex <- iso_get_reader_example("dual_inlet_example.did") %>%  
  iso_read_dual_inlet(quiet = TRUE)
# access binary
bin <- ex$binary
# use structure mapping
bin %>%
  isoreader:::move_to_pos(1340) %>%
  isoreader:::map_binary_structure(length = 200)
#> # Binary data structure:  
#> 0001340: {4d 65 73 73 61 67 65 44 61 74 61}
#> 0001351:   <etx>
#> 0001355:   {/}<fef-0><fef-12>{Peak Center foun..}<4x00><1-000><fef-1c>{Peak Center found at [61032]}<fef-09>{CUserInfo}<ffff>{00 00 00 00 1b 80}
#> 0001501: <etx>
#> 0001505: {/}<fef-0><fef-12>

This structure representation shows recognized control elements in <...> and data elements in {...} which are converted to text or numeric representation if the interpretation is unambiguous, or plain hexadecimal characters if the nature of the data cannot be determined with certainty. Because this function tries all possible control elements and data interpretations, it is quite slow and may take a while if run for large stretches of binary code (i.e. if the length parameter is very long).

For an overview of all the control elements that are currently consider, use the internal get_ctrl_blocks_config_df() function.

isoreader:::get_ctrl_blocks_config_df()
#> # A tibble: 26 × 3
#>    block    regexp                              hexadecimal                     
#>    <chr>    <chr>                               <chr>                           
#>  1 del-nl   "\x7f\x85"                          7f 85                           
#>  2 eop-nl   "܅"                                 dc 85                           
#>  3 0b-80    "\v\x80"                            0b 80                           
#>  4 e0-81    "\xe0\x81"                          e0 81                           
#>  5 ce-80    "\u0380"                            ce 80                           
#>  6 ce-8a    "Ί"                                 ce 8a                           
#>  7 ee-85    "\xee\x85"                          ee 85                           
#>  8 75-84    "u\x84"                             75 84                           
#>  9 ff-80    "\\x00\xff\x80\\x00"                5c 78 30 30 ff 80 5c 78 30 30   
#> 10 07-80-id "\x05\x80.\xff(\\x00|\x80|\xff){2}" 05 80 2e ff 28 5c 78 30 30 7c 8…
#> # … with 16 more rows

Additional information can be gleaned from the so-called control blocks, which are larger structural elements of Isodat binary files and are kept in a data frame within the binary object (again only available in debug mode).

bin$C_blocks
#> # A tibble: 107 × 5
#>    id1   id2   block                 start   end
#>    <chr> <chr> <chr>                 <int> <dbl>
#>  1 06    0b    CFileHeader               1    17
#>  2 01    0b    CTimeObject             182   198
#>  3 02    04    CStr                    241   250
#>  4 01    0a    CDataIndex              471   486
#>  5 01    11    CSeqLineIndexData       513   535
#>  6 03    05    CData                   588   598
#>  7 01    13    CDualInletBlockData    1133  1157
#>  8 01    10    CMeasurmentInfos       1240  1261
#>  9 01    15    CISLScriptMessageData  1324  1350
#> 10 01    11    CMeasurmentErrors      1945  1967
#> # … with 97 more rows

Same as for specific byte positions, one can use the control blocks to navigate the file and map_binary_structure.

bin %>%
  isoreader:::move_to_C_block("CMethod") %>%
  isoreader:::map_binary_structure(length = 200)
#> # Binary data structure:  
#> 0080269: <etx>
#> 0080273:   {-}<fef-0><fef-06>{Method}<4x00>
#> 0080299:   <stx>
#> 0080303:     <7-000>
#> 0080307:     <C-01-09 CMolecule>
#> 0080322:   <etx>
#> 0080326:   {-}<fef-0d>{[email protected]}<fef-0d>{[email protected]}<4x00><1-000><fef-03>{CO2}
#> 0080406:   <C-00-0b CPartMirror>
#> 0080423:   <C-02-13 CMethodPrintoutDesc>
#> 0080448: <etx>
#> 0080452: {-}<fef-0><fef-0><4x00>
#> 0080466: <stx>