Data Sources

Figure 1: Prompt: Can you draw me an anime style-d picture of a computer wielding a machete?

“I have sat in many meetings in which people talk about data as if it were a magical substance that would solve everyone’s problems if only they had more of it.” - Jed Sundwall (Sundwall, n.d.)

There are two primary lenses I reach for when you start talking to me about data:

• The friction associated with being a data scientist, both in terms of the technical execution of a workflow and the mass moving data can entail
• Garbage in - garbage out

To address the first, I’ve spent a great deal of time learning, observing, teaching, relearning, and documenting my pathway towards reproducible and more durable knowledge and useful notes. Addressing the second requires a bit of a shift in perspective. Though it’s a common modeling quote that “All models are wrong, some are useful” - George Box Box (1976), this doesn’t usually suffice when the ask-r has that favorite dataset or model they want to see used in an analysis, and while annoying there is (usually) less friction for all involved if you’ve done all the heavy lifting of functionalizing your analysis properly. In theory that should be as easy as swapping inputs…

That being said, there are diminishing returns on all of that effort. Some datasets are more authoritative or more widely used, and so there is value added in quantifying skill using that more consistent benchmark. More tangibly, some datasets are formatted and distributed using modern computational patterns and formats, and that ease of accessibility tips the scales heavily in favor of using that data, since the inertia and friction of that data access has been at least partially mitigated by the provider, meaning the limiting factor to scaling your domain is the size of your PC (or the credit limit on the card tied to your cloud budget).

An additional and very frustrating piece of friction I encounter almost daily is that the chasm between theory and practice has grown so large that the “head work” of swapping inputs and the “hand work” of actually implementing and executing that pipeline can be quite significant, particularly when dealing with spatial scale. Below I’ve structured a little data catalog layout that more closely documents and follows the different datasets I encounter and deploy.

Data Sources

Parameterizeing earth system models typically requires a rich mosaic of input data. I’ve constructed my own rough tagging system for the ones I use or encounter most commonly; use the categories on the sidebar to the left to filter the grid below. They don’t follow NASA’s [[20240507073418]] Data Processing Levels but are instead intended to provide overarching themes to categorize them. Everything I point to in this page is scientific grade in the sense that it has a citation and has or could be used to make a defensible decision, and of course these categorizations all bleed into each other at the edges. If you’re interested in paramitizing the system for yourself, check out the metrology page.

TipA preamble: Setting up access and passwords

You can thank Papa Bush for opening up the satellite archives to the general public, but open and free does not mean frictionless and several of the more authoritative data warehouses make you sign up for an account. Additionally, if you’d like to use the climateR package you’ll need to place credentials into a file so that you can hit these resources..

I am a security disaster and the easiest way to make your life simple and ensure that anyone can use your account is to create the same login for all services. An “ideal” password that works across all services is something shorter with a capital letter and a few numerics at the end.

```{r}
remotes::install_github("mikejohnson51/climateR",force = TRUE)
climateR::writeNetrc(login = "user",password = "Password123")
climateR::checkNetrc()
sites_stack <- climateR::getGLDAS(AOI=AOI::aoi_get(state = "OR"),
                        varname='rootmoist_inst',
                        model='NOAH025_M.2.1',
                        startDate='2006-10-01',
                        endDate='2007-09-30',
                        verbose = TRUE)
```

Datasets

Reminder: use the categories on the sidebar to the left to manipulate the grid view below

CautionUnder refinement

This is being reworked, several placeholders and defaults are still present.

Aside: datasets not included here

Given that some of my motivation in using data is to put food on my family, sometimes a popular dataset the community latches onto is unavalible for my use case. While this doesn’t mean it’s a bad dataset, it’s not going to show up there so to document that list because I’ve had questions:

• FABDEM: Elevations instantiated with Copernicus, but has an overly restrictive (non-commercial) reuse [[20240824130820|Software license]].
• Dewberries RAS stacmap: has the MCAT-RAS cataloged outputs from the FEMA MIP, but source HEC-RAS files are all held in private buckets.

Explainers

Box, George E. P. 1976. “Science and Statistics.” Journal of the American Statistical Association 71 (356): 791–99. https://doi.org/10.1080/01621459.1976.10480949.

Sundwall, Jed. n.d. We Don’t Talk About Open Data. Https://radiant.earth/blog/2023/05/we-dont-talk-about-open-data/.