Bounded variant of DESCRIPTION() for callers that need to keep any single
call's work small (e.g. to stay under a network-layer request timeout),
rather than materialising every match in one unbounded scan.
Usage
DESCRIPTION_CHUNK(
pattern,
type = getOption("codeminer.code_type"),
cursor = 0L,
batch_size = getOption("codeminer.chunk_batch_size", default = 200000L),
total_rows = NULL,
accumulated_so_far = 0L,
max_rows = getOption("codeminer.max_leaf_rows", default = 30000L),
max_chunk_matches = getOption("codeminer.max_chunk_matches", default = 20000L),
lookup_version = getOption("codeminer.lookup_version", default = "latest"),
ignore_case = TRUE,
preferred_description_only = TRUE,
col_filters = "default"
)Arguments
- pattern
The description to search for. See
stringr::str_detect()for details.- type
character. Type of clinical code system to be searched. Optional if input is a data frame with code_type column. Depends on what is available in the lookup tables. See
add_lookup_table()on how to add new lookup tables. This can also be configured through thecodeminer.code_typeoption.- cursor
Integer.
rowidto start this chunk's scan from.0for the first call.- batch_size
Integer. Number of
rowids to scan in this call. Defaults togetOption("codeminer.chunk_batch_size", default = 2000).- total_rows
Integer or
NULL. The underlying table's total row count. PassNULLon the first call (it will be resolved and returned); pass the previously-returned value on subsequent calls to skip re-resolving it.- accumulated_so_far
Integer. Total matched rows found by this leaf's previous chunks (before this one).
0for the first call. Used to checkmax_rowscumulatively across the whole chunked fetch, not just this one call.- max_rows
Integer. Ceiling on this leaf's cumulative matched rows across all chunks. Aborts with class
codeminer_max_leaf_rows_exceededif exceeded. Defaults togetOption("codeminer.max_leaf_rows", default = 30000).- max_chunk_matches
Integer. Ceiling on the number of codes matched within this single chunk, checked before the matches are expanded via
CODES(). Guards against a chunk whose scan range happens to match densely -CODES()'s cost scales with match count, not rows scanned, so this bounds a single call's worst case independently ofbatch_size. Aborts with classcodeminer_chunk_match_limit_exceededif exceeded. Defaults togetOption("codeminer.max_chunk_matches", default = 20000).- lookup_version
character. Version of the lookup table to use. Default:
"latest". Can be configured through thecodeminer.lookup_versionoption.- ignore_case
If
TRUE(default), ignore case indescription.- preferred_description_only
logical. IfTRUE(default), return only preferred descriptions.- col_filters
Column filters to apply. See
CODES()for details.
Value
A list with result (a codeminer_codelist of this chunk's
matches), next_cursor (integer, pass back as cursor next call),
total_rows (integer, pass back as total_rows next call), and
exhausted (logical — TRUE once next_cursor has covered the whole
table).
Details
Each call scans only the rowid range [cursor, cursor + batch_size) of
the underlying lookup table for matches, expands those matches to full
rows via CODES(), and returns a cursor for the next chunk. Callers
should keep calling with the returned next_cursor (and the same
total_rows and accumulated_so_far, updated each time) until
exhausted is TRUE.
See also
Other Clinical code lookups and mappings:
CODES(),
CODES_ALL_CHUNK(),
CODES_LIKE_CHUNK(),
MAP(),
get_lookup_table(),
get_mapping_table(),
get_relationship_table(),
get_relationship_tree()
Examples
create_dummy_database()
#> ✔ Dummy database ready to use!
#> ℹ To reconnect to your previous database:
#> `Sys.setenv(CODEMINER_DB_PATH = "/tmp/Rtmp2UzWkJ/file1c05658d538.duckdb")`
#> `codeminer_connect()`
chunk <- DESCRIPTION_CHUNK("cyst", type = "ICD-10", batch_size = 100)
#> ℹ Using "UKB v4" as the latest lookup version for "ICD-10".
chunk$result
#> <codeminer_codelist>: 0 codes
#>
#> # A tibble: 0 × 3
#> # ℹ 3 variables: code <chr>, description <chr>, code_type <chr>
chunk$exhausted
#> [1] FALSE