diff --git a/.Renviron.example b/.Renviron.example new file mode 100644 index 0000000..ea9bb23 --- /dev/null +++ b/.Renviron.example @@ -0,0 +1,2 @@ +AZ_STORAGE_EP=[your azure blob storage endpoint url] +AZ_TABLE_EP=[your azure table storage endpoint url] diff --git a/R/get_auth_token.R b/R/get_auth_token.R index 6d61d41..a71c438 100644 --- a/R/get_auth_token.R +++ b/R/get_auth_token.R @@ -18,8 +18,8 @@ #' to help provide an appropriate string or vector. #' If setting version to 2, ensure that the `aad_version` argument is also set #' to 2. Both are set to use AAD version 1 by default. -#' @param tenant A string specifying the Azure tenant. Defaults to -#' `"common"`. See [AzureAuth::get_azure_token] for other values. +#' @param tenant A string specifying the Azure tenant. Defaults to `"common"`. +#' See [AzureAuth::get_azure_token] for other values. #' @param client_id A string specifying the application ID (aka client ID). If #' `NULL`, (the default) the function attempts to obtain the client ID from the #' Azure Resource Manager token, or prompts the user to log in to obtain it. diff --git a/R/list_files.R b/R/list_files.R index 909a9f7..f68572c 100644 --- a/R/list_files.R +++ b/R/list_files.R @@ -28,7 +28,7 @@ list_files <- function(container, dir = "", ext = "", recursive = FALSE) { stopifnot(rlang::is_character(c(dir, ext), 2)) stopifnot(rlang::is_bool(recursive)) - pnf_msg <- ct_error_msg("Path {.val {path}} not found") + pnf_msg <- ct_error_msg("Path {.val {dir}} not found") check_that(dir, \(x) AzureStor::blob_dir_exists(container, x), pnf_msg) ext_rx <- ifelse(nzchar(ext), gsub("^\\.+", "\\.", ext), ".*") # nolint @@ -40,7 +40,7 @@ list_files <- function(container, dir = "", ext = "", recursive = FALSE) { if (nrow(tbl) == 0) { fix_path <- \(p) sub("^/+$", "", sub("^([^/])(.*)", "/\\1\\2", p)) # nolint ext <- if (nzchar(ext)) paste0(" ", ext) - msg <- "No{ext} files found in {.val [{container$name}]:{fix_path(path)}}" + msg <- "No{ext} files found in {.val [{container$name}]:{fix_path(dir)}}" cli::cli_alert_info(msg) invisible(character(0)) } else { diff --git a/README.md b/README.md index ce12572..b4da4fb 100644 --- a/README.md +++ b/README.md @@ -19,12 +19,13 @@ blob and table storage and reading in data from files. ## Status -The package is in development. -Please [create an issue][issues] if you have ideas for its improvement. +The package is stable and useable. +Please [create a GitHub issue][issues] if you encounter problems, or have ideas +for its improvement. ## Installation -You can install the development version of `{azkit}` with: +You can install `{azkit}` with: ``` r # install.packages("pak") @@ -36,17 +37,25 @@ pak::pak("The-Strategy-Unit/azkit") A primary function in `{azkit}` enables access to an Azure blob container: ```r +# This assumes you have Azure authentication working, and the correct +# Azure endpoint variables in your environment (see below). +# NB the file/data locations used here are invented examples. data_container <- azkit::get_container("data-container") ``` -Authentication is handled automatically by `get_container()`, but if you need -to, you can explicitly return an authentication token for inspection or re-use: +Authentication should be handled automatically by `get_container()`, but if you +need to, you can explicitly return an authentication token for inspection or +re-use: ```r my_token <- azkit::get_auth_token() - ``` +(For issues with authentication, including mysterious error messages, please +see the Troubleshooting section below or the [Troubleshooting vignette][trblv].) + +[trblv]: https://the-strategy-unit.github.io/azkit/articles/troubleshooting.html + ```r data_container <- azkit::get_container("data-container", token = my_token) ``` @@ -63,7 +72,6 @@ For example: ```r pqt_data <- azkit::read_azure_parquet(data_container, "important_data.parquet") - ``` To read in any file from the container in raw format, to be passed to the @@ -73,6 +81,8 @@ handler of your choice, use: raw_data <- azkit::read_azure_file(data_container, "misc_data.ext") ``` +Currently these functions only read in a single file at a time. + You can map over multiple files by first using `azkit::list_files()` and then passing the file paths to the `read*` function: @@ -81,12 +91,10 @@ azkit::list_files(data_container, "data/latest", "parquet") |> purrr::map(\(x) azkit::read_azure_parquet(data_container, x)) ``` -Currently these functions only read in a single file at a time. - You can also pass through arguments in `...` that will be applied to the appropriate handler function (see documentation). For example, `readr::read_delim()` is used under the hood by -`azkit::read_azure_csv`, so you can pass through a config argument such as +`azkit::read_azure_csv`, so you can pass through an argument such as `col_types`: ```r @@ -97,7 +105,7 @@ csv_data <- data_container |> ## Environment variables -To facilitate access to Azure Storage you may want to set some environment +To facilitate access to Azure Storage you should set some environment variables. The neatest way to do this is to include a [`.Renviron` file][posit_env] in your project folder. @@ -120,6 +128,16 @@ AZ_TABLE_EP= Azure authentication is probably the main area where you might experience difficulty. +> [!NOTE] +> Users at The Strategy Unit: when initially authenticating with Azure, you +> will be sent to the web. +> Since you may be logged into one of multiple Microsoft tenants, it is +> important that you authenticate with the same account (MLCSU) that you use +> for access to Azure storage. +> If you authenticate when logged into your nhs.net Microsoft account in your +> browser, your token will not be useable for access to Azure. + + To debug, try running: ```r diff --git a/man/get_auth_token.Rd b/man/get_auth_token.Rd index cadf4bc..46f14dc 100644 --- a/man/get_auth_token.Rd +++ b/man/get_auth_token.Rd @@ -26,8 +26,8 @@ to help provide an appropriate string or vector. If setting version to 2, ensure that the \code{aad_version} argument is also set to 2. Both are set to use AAD version 1 by default.} -\item{tenant}{A string specifying the Azure tenant. Defaults to -\code{"common"}. See \link[AzureAuth:get_azure_token]{AzureAuth::get_azure_token} for other values.} +\item{tenant}{A string specifying the Azure tenant. Defaults to \code{"common"}. +See \link[AzureAuth:get_azure_token]{AzureAuth::get_azure_token} for other values.} \item{client_id}{A string specifying the application ID (aka client ID). If \code{NULL}, (the default) the function attempts to obtain the client ID from the diff --git a/vignettes/azkit.qmd b/vignettes/azkit.qmd index 3c97b13..91c10b0 100644 --- a/vignettes/azkit.qmd +++ b/vignettes/azkit.qmd @@ -8,271 +8,358 @@ knitr: opts_chunk: collapse: true comment: '#>' -execute: +execute: eval: false -editor_options: +editor_options: chunk_output_type: console --- `{azkit}` is an R package to help you: * handle authentication with Azure -* perform basic tasks accessing blob (unstructured data) and table storage -(structured data) -* read in data from some common file types. +* perform basic tasks accessing Azure blob storage and table storage -The purpose of this page is to show you a few ways you might use {azkit} functions -to achieve certain tasks. +The purpose of this page is to show you a few ways you might use {azkit} +functions to achieve certain tasks. It's not the only way you can set things up and use the functions, it's just an opinionated example workflow - you may choose to do things differently 😀. -## Setting up security credentials +## Authentication with Azure storage -> 💡 Tip -> This only needs to be done once but if you choose to do this for a project rather -than globally (user) then every project will need credentials set up. +This is an important first step. -Azure requires credentials set up locally on your computer in order to the data -it contains. There are a number of places this information can -be stored and their use depends on your team's use and the security level -required. +`{azkit}` supports various approaches to authentication and authorisation, +depending on the kind of `resource` you need to access. -### `.Renviron` files +The `get_auth_token()` function therefore supports various options, for example +whether to generate a v1 or v2 token, whether the code is running on a managed +resource or locally, whether to use a locally-stored secret or an authorisation +code, and so on. +Our function is a wrapper around functions in the `{AzureAuth}` package, and +reading the documentation provided for that package is a good way to explore +what specific approaches might be required for specific circumstances. -There are two `.Renviron` files, one can be stored in your project folder and -one can be set globally so that any project can access the credential. Both files -need to be created and set up and can either be done manually or, more -conveniently, through using the `usethis` package: +For standard users, however, the default options provided by our function should +be sufficient and should be the simplest way to get up and running. -> âš ī¸ Important -> Don't forget to ensure that the `.gitignore` has `.Renviron` listed. This is -particularly important if the file is saved in the project. +As a local user (not running on a managed resource), you first of all need to +get an authentication token that verifies to Azure that you have a certain user +account. -```{r} -# creates or opens an existing file in the project -usethis::edit_r_environ(scope = "project") -# Add a file to .gitignore if you haven't already done so -usethis::use_git_ignore(".Renviron") +Do this from your computer when online, as it requires user interaction - it +will require you to log in via your web browser, or via a Microsoft login app. +R cannot do this bit for you 😊. -# creates or opens an existing file in the global -usethis::edit_r_environ(scope = "user") +Run -# just running the following defaults to "user" -usethis::edit_r_environ() +```r +azkit::get_auth_token() ``` -When opening a `.Renviron` file for the first time, whether user or project, it -will be blank and the following needs to be added: +at the R console. Alternatively, install the [Azure CLI tool][azcli] and do: +```bash +az login ``` -AZ_STORAGE_EP= + +at the terminal. +This should pop up a window asking you to authenticate with Azure online - +beware, sometimes this window is not immediately visible, it may be hidden +behind other windows on your screen. + +The authentication token will be stored under your user account files on your +computer. + +You can see which tokens you have by running + +```r +AzureAuth::list_azure_tokens() ``` -and after the `=` sign add the URL of your Azure endpoint (with no quotes). -Any changes to the files will require restart in R and if you use the usethis -function it will remind you in a message that appears in the Console. -> 💡 Tip -> If you use both `.Renviron` files the project will be read before the user. -Ensure that the project `.Renviron` is not completely blank though as this -will default to blank even if the details are in the user file. +See also the [Troubleshooting vignette][trbv]. + +[azcli]: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli +[trbv]: https://the-strategy-unit.github.io/azkit/articles/troubleshooting.html -## Using the `keyring` package for credentials -The package {keyring} can be a good way to separate credentials and are -also available globally so any project can access them. +### Refreshing your auth token -To set the credential: +In a data analysis workflow, if you have done something like: -```{r} -library(keyring) -keyring::key_set("AZ_STORAGE_EP") +```r +token <- azkit::get_auth_token() ``` -A pop up box will appear where you can copy the url (without quotes). To -retrieve the credential you will need the following code: +and the token later expires*, you should be able to just do this again: + +```r +token <- azkit::get_auth_token() +``` +but alternatively you can use -```{r} -keyring::key_get("AZ_STORAGE_EP") +```r +azkit::refresh_token(token) ``` -To view what keyrings you have: +If these don't work, you can force azkit to go get you a whole new fresh token +from the internet by doing: -```{r} -keyring::key_list() +```r +token <- get_auth_token(force_refresh = TRUE) ``` -## Find what containers there available to view +\* auth tokens have a limited validity period but are supposed to auto-refresh +when needed -```{r} -library(azkit) + +## Setting up environment variables + +Use an `.Renviron` file to set 2 required environment variables (envvars). +These are the endpoint URLs of your Azure storage account, for blob storage and +table storage. + +There are two ways you can set variables using `.Renviron` files: + +* Globally for your local (eg Windows) user account +* Project-specific + +The `{usethis}` packages provides a handy helper to create/edit these files: + +To use the first, global option (probably the most straightforward): + +```r +usethis::edit_r_environ() ``` -To see the containers you have access to: +or to edit a project-specific `.Renviron` file, while within the project folder: -```{r} -# This will work if you have the AZ_STORAGE_EP url stored in an `.Renviron` file -azkit::list_container_names() +```r +usethis::edit_r_environ(scope = "project") +``` + +The latter makes sense if you have multiple projects that may be using different +Azure endpoints, or a specific project that uses a different account to +your usual one. +Otherwise it is probably simpler to set the variables globally and then you can +basically forget about it! -# The default is the same as writing regardless of which `.Renviron` file is used -azkit::list_container_names(Sys.getenv("AZ_STORAGE_EP")) +> âš ī¸ **Important** +> +> If using a project-specific file, ensure this is listed in your `.gitignore` +> file + +### What to add to your `.Renviron` file + +For the purposes of this package, the file just needs to contain two endpoint +URLs with the following names: -# If you are using keyring you will need to type out the code: -azkit::list_container_names(endpoint_url = keyring::key_get("AZ_STORAGE_EP")) ``` +AZ_STORAGE_EP=[your azure blob storage endpoint url] +AZ_TABLE_EP=[your azure table storage endpoint url] +``` + +See the `.Renviron.example` file in the azkit GitHub repository. + +For use within The Strategy Unit, ask a member of [the Data Science team][suds] +to send you the necessary values for these URLs. + +[suds]: https://the-strategy-unit.github.io/data_science/about.html + +Changes to the `.Renviron` file will require you to reload your environment in +order to update it, such as by restarting your R session. + +`azkit` is expecting these variables to be present in your working environment. +The reason we set these as environment variables is that the URLs are generally +not meant to be made public, for security reasons. +Referencing them as envvars means that their actual values do not need to be +included explicitly in any code that might be made public. + + +> 💡 **Tip** +> +> If you use a project `.Renviron` file as well as a global one, the variables +> in the project file will override those in the global file. +> Ensure that any project `.Renviron` is not completely blank, as this +> will unset any variables from the global file. + + -## Accessing a container +## Working with Azure blob storage -Two functions are used together to access files from a particular container. +### Containers -> 💡 Tip -> The following will refer to `"supporting-data"` and this is a placeholder so -will need to be changed to a container you have access to. +Reading in data from Azure blob storage requires you to first set up access to +a particular `container` where your data is located. -The function `list_files()` only returns files according to their type, rather -than individual files which may be more familiar with which list _all_ files: +The `azkit::get_container()` function makes this easy. -```{r} -# library(here) +You may have a container name stored as an environment variable, for security. +If not, just provide the name of the container as a string instead. -# This is just an example of a base R function that returns all the files -# list.files(here::here()) +First read this in and then use it to create a container object in R: + +```r +container_name <- Sys.getenv("AZ_STORAGE_DATA_CONTAINER") +# or as a string: +# container_name <- "example-data" +data_container <- azkit::get_container(container_name) ``` -`list_files()` requires `get_container()` to point to the exact container and -provide the relevant credentials: +If you don't know the names of the containers you have access to, you can use: -```{r} -# Retrieving the container information will be used in subsequent code -sd_container <- get_container( - container_name = "supporting-data", - endpoint_url = Sys.getenv("AZ_STORAGE_EP") -) +```r +azkit::list_container_names() ``` +to output a list. -```{r} -# "supporting_data" is a name that you will have to change according to what the -# containers you have access to and what they are called -azkit::list_files( - container = sd_container, - ext = "csv" -) +> 💡 **Tip** +> +> Alternatively, it can be useful to browse your Azure storage account on the +> web at or use [Azure Storage Explorer][aseapp]. + +[aseapp]: https://azure.microsoft.com/en-us/products/storage/storage-explorer/ + + +### Accessing files in the container + +Use `azkit::list_files()` to see what files are available. + +Re-using our `data_container` variable from the code above, we might do +something like: + +```r +azkit::list_files(data_container, "my-data-folder") ``` -If the container has sub folders the `list_files()` can also return the files -in those when the `recursive =` parameter is set to TRUE (the default is FALSE): +This will return a list of the all the files in that specific directory. +By default this is non-recursive: it does not return any files within +sub-directories. +You can change this behaviour by using the `recursive` argument, but beware that +you might end up with a lot of filenames if there is lots of data stored in a +sub-directory structure! -```{r} -azkit::list_files( - container = sd_container, - ext = "csv", - recursive = TRUE -) +```r +azkit::list_files(data_container, "my-data-folder", recursive = TRUE) ``` -> 💡 Tip -> If you'd prefer to view Azure in a GUI (Graphical User Interface) and have -permissions use the Microsoft program Azure Storage Explorer to explore -containers or access via a browser by logging into -[https://portal.azure.com/#home](https://portal.azure.com/#home) +You can also limit `list_files` to only return files with a certain filetype +extension (no `.` required before the extension). +(By default it will list all files.) -## Reading a file +```r +azkit::list_files(data_container, "my-data-folder", ext = "json") +``` -For all the functions to get particular files, they need to be used in -conjunction with the `get_container()` function that provides details of the -container and credentials. +Once you have your filename, you can use one of a selection of azkit functions +to help read the data into R. -For the simplest access downloading a csv: +azkit comes with five functions optimised for reading in data from: -```{r} -# Note this code is "nested" so the get_container() function is inside -# the read_azure_csv() function +* parquet (`azkit::read_azure_parquet()`) +* json (`azkit::read_azure_json()`) +* json.gz (`azkit::read_azure_jsongz()`) +* csv (`azkit::read_azure_csv()`) +* rds (`azkit::read_azure_rds()`) + +These come with helper functions already set up to read the data into R in the +most convenient way. +There is also scope to customise how these functions operate - see each +function's help page for more. + +For example, `azkit::read_azure_csv()` allows you to pass additional options +through to `readr::read_delim`, such as the `col_types` argument. + +There is also the `azkit::read_azure_file()` function that will attempt to read +from any given file, for situations not covered by the above 5. +But you will need to handle the output returned by this yourself. -azkit::read_azure_csv( - container = sd_container, - file = "mitigator-lookup.csv" -) -``` -You can also use other arguments that will be applied to the specific file -reader through the argument `...`. For example, -`readr::read_delim()` is used by the function `read_azure_csv` and because of -this it's possible to use other arguments that the `readr::read_delim()` could -use: - - -```{r} -# Note this code is "piped" so the get_container() function is before the -# the read_azure_csv() function - -azkit::get_container( - container_name = "supporting-data" -) |> - azkit::read_azure_csv( - file = "mitigator-lookup.csv", - # select specific columns and can be used with other dplyr functions - col_select = dplyr::starts_with("active") - ) +#### Example: reading a parquet file + +```r +parquet_data <- azkit::read_azure_parquet(data_container, "data/info.parquet") ``` -Functions for other file types all follow the same pattern using `get_container()` -and detailing the file wanted: -```{r} -azkit::read_azure_rds() -azkit::read_azure_json() -azkit::read_azure_jsongz() -azkit::read_azure_parquet() +#### Example: reading a CSV file + +Simplest method: + +```r +csv_data <- azkit::read_azure_csv(data_container, "csv_data/info.csv") ``` -These all will download in formats that you will be familiar with but if the -data is required in the `raw binary` form that is stored in Azure and then -parsed/translated by some other means then the following function will retrieve -any data type: +With additional options passed through: + +```r +csv_data <- data_container |> + azkit::read_azure_csv("csv_data/info.csv", col_types = "cc-ii") +``` -```{r} -# This is not a human readable form! -azkit::read_azure_file( - container = sd_container, - file = "mitigator-lookup.csv" +```r +csv_data <- azkit::read_azure_csv( + data_container, + "csv_data/info.csv", + col_select = tidyselect::starts_with("active") ) ``` -This makes transferring data very fast and can be translated locally. In this -example from a .csv it would look like: +#### Example: reading multiple files -```{r} -raw_data <- azkit::read_azure_file( - container = sd_container, - file = "mitigator-lookup.csv" -) +The `azkit::read_azure_*` functions don't read in multiple files by default, but +this can be achieved by using them within an apply or map operation: + +```r +csv_files <- azkit::list_files(data_container, "csv_data", ext = "csv") +csv_names <- tools::file_path_sans_ext(basename(csv_files)) +csv_data_list <- csv_files |> + purrr::map(\(x) azkit::read_azure_csv(data_container, x)) |> + rlang::set_names(csv_names) # optional, may be useful +``` -csv_data <- read.csv(text = rawToChar(raw_data)) -# view top few rows of what is now a data frame -head(csv_data) +## Working with Azure table storage + +Table storage does not require access to containers. +You just need the name of the table you want to access. + +We will use the `azkit::read_azure_table()` function. + +You may have a table name stored as an environment variable, for security. +If not, just provide the name of the table as a string instead. + +To read in the entire table: + +```r +table_name <- Sys.getenv("AZ_TABLE_NAME") +# or as a string: +# table_name <- "example-table" +table_data <- azkit::read_azure_table(table_name) ``` -## Reading multiple files +Within this function, you can filter the table that is returned by using OData +clauses ("system query options"). +This is especially useful with large tables where you wish to improve speed by +only requesting certain rows or columns. +Use the `filter`, `select`, and/or `top` arguments to achieve this. -The functions don't read in multiple files but can be combined with other code -to do this using loops or the equivalent using the purrr package: +### Example of the OData syntax -```{r} -list_of_csvs <- azkit::list_files( - container = sd_container, - ext = "csv" +```r +table_data <- azkit::read_azure_table( + table_name, + filter = "Status eq 'Active' and Score ge 10", # filter rows by variable value + select = "PartitionKey,RowKey,Status,Score", # only return certain columns + top = 100 # only return the first 100 rows ) - -list_of_csvs |> - # get the name from the file name but drop the extension detail - purrr::set_names(basename(tools::file_path_sans_ext(list_of_csvs))) |> - purrr::map(\(x) { - read_azure_csv( - container = sd_container, - x - ) - }) |> - list2env(.GlobalEnv) ``` + +A very initial guide to how to construct your own clauses (query options) can be +found on [Microsoft Learn][odata]. +Although the examples there are not written in R, the syntax for the query +options should be in a valid format for what you need to supply to +`azkit::read_azure_table()`. + +[odata]: https://learn.microsoft.com/en-us/odata/concepts/queryoptions-overview diff --git a/vignettes/troubleshooting.qmd b/vignettes/troubleshooting.qmd index 0259cdc..1598e38 100644 --- a/vignettes/troubleshooting.qmd +++ b/vignettes/troubleshooting.qmd @@ -11,55 +11,64 @@ knitr: eval: false --- -## Troubleshooting - Azure authentication is probably the main area where you might experience difficulty. -To debug, try running: - -```{r} -library(azkit) +To debug initially, try running: +```r token <- azkit::get_auth_token() ``` -It should look like: -``` +It should look something like: + +```sh Azure Active Directory v1.0 token for resource https://url.com - Tenant: - App ID: - Authentication method: - Token valid from: to: - MD5 hash of inputs: + Tenant: + App ID: + Authentication method: + Token valid from: to: + MD5 hash of inputs: ``` -If this errors, it's worth testing the authentication is set up correctly by +If this errors, it's worth testing the authentication is set up correctly by using another package: -```{r} -# install.packages("AzureRMR") +```r +# pak::pak("AzureRMR") -# This will open the browser and prompt for a Microsoft Account to be used to connect +# This will open the browser and prompt for a Microsoft account to be used to +# connect AzureRMR::create_azure_login() - AzureRMR::get_azure_login() ``` A successful authentication will result in the following message: -``` +```sh Loading Azure Resource Manager login for default tenant ``` + +> â„šī¸ **Note** +> +> Users at The Strategy Unit: when initially authenticating with Azure, you +> will be sent to the web. +> Since you may be logged into one of multiple Microsoft tenants, it is +> important that you authenticate with the same account (MLCSU) that is used +> for access to Azure storage. +> If you authenticate when logged into your nhs.net Microsoft account in your +> browser, your token will not be useable for access to Azure. + + If you had successfully used azkit to get a token and then run the code above you may now have multiple tokens which you can view: -```{r} +```r AzureRMR::list_azure_tokens() ``` To refresh a token: -```{r} +```r azkit::refresh_token(token) ``` @@ -68,7 +77,11 @@ azkit::refresh_token(token) If you get errors when reading in files, first check that you are passing in the full and correct file path relative to the root directory of the container. -Next try reading in the raw data with `read_azure_file()` which returns the binary -form of data (which isn't human readable). If this is successful then you will -be able to pass the data to a handler function that's relevant to the data. -Details can be found in [Getting Started](./azkit.html). +You may wish to use the Azure Portal website or the Azure Storage Explorer app +to check that you have a valid container name and file path. + +The following function should also return `TRUE` if the file exists: + +```r +AzureStor::blob_exists(container_name, file) +```