Compute
While I am a patient man and am happy to wait, many objective of data science care about speed. If that’s your jam, check out https://duckdblabs.github.io/db-benchmark/, but as quoted from the duckdblabs page: “We also include the syntax being timed alongside the timing. This way you can immediately see whether you are doing these tasks or not, and if the timing differences matter to you or not.” (Database-Like Ops Benchmark, n.d.)
The complications of scaling and compute
Many user and use cases desire high resolution data, but that use can become a significant cost burden the finer in resolution you move. Furthermore, this cost increase does not correspond with the changes in resolution (i.g. a process at 10 meters will not take twice as long as a 20 meter one). What’s often missing in this conversation is some geographic context and some bounds around the scales of the process you’re modeling. Here’s that breakdown with considerations towards processing speed and size concerns.
Local
The most accessible way to run calculations is on the computer right in front of your face. However, how you set up and run your computer (environment) might vary wildly from someone else, and so issues like reproducibility and replicability creep in as you advance. One of the most direct ways to overcome some of those hurdles is to create a “virtual environment” within which you develop and test your process. While there are all sorts of different virtual environments that you can use that provide all the bells and whistle you might want, those can also be very “heavy” both in terms of the amount of disk space the environment takes up and how ineffectively the environment uses your hardware. When performance and ease of scaleability matter, Docker is a great way to keep a task reproducible which makes porting and scaling your workflow almost frictionless. Per the Docker landing page: “Docker helps developers build, share, run, and verify applications anywhere — without tedious environment configuration or management.” See how I install and use Docker for tips and cheatsheets.
Cloud
ARCO resource
https://radiantearth.github.io/stac-browser/#/
References
Technical Readiness Levels
The pathway to operations has two major complications, the technical and the societal. The technical includes aspects such as speed and scalability, whereas the social is dependent on attributes such as explainibility and the deviation from baseline.
NOAA READINESS LEVELS Readiness Levels are a systematic project metric or measurement system that supports assessments of the maturity of R&D projects for transition from research to operation, application, commercial product or service, or other use and allows the consistent comparison of maturity between different types of R&D projects.
Readiness Level 1
Basic research, experimental or theoretical work undertaken primarily to acquire new knowledge of the underlying foundations of phenomena and observable facts, without any particular application or use in view. Basic research can be oriented or directed towards some broad fields of general interest, with the explicit goal of a range of future applications (OECD, 2015).
Readiness Level 2
Applied research, original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily towards a specific, practical aim or objective. Applied research is undertaken either to determine possible uses for the findings of basic research, or to determine new methods or ways of achieving specific and predetermined objectives (OECD, 2015).
Readiness Level 3
Proof-of-concept for system, process, product, service, or tool; this can be considered an early phase of experimental development; feasibility studies may be included.
Readiness Level 4
Successful evaluation of system, subsystem, process, product, service, or tool in a laboratory or other experimental environment; this can be considered an intermediate phase of development.
Readiness Level 5
Successful evaluation of system, subsystem process, product, service, or tool in relevant environment through testing and prototyping; this can be considered the final stage of development before demonstration begins.
Readiness Level 6
Demonstration of a prototype system, subsystem, process, product, service, or tool in relevant or test environment (potential demonstrated).
Readiness Level 7
Prototype system, process, product, service or tool demonstrated in an operational or other relevant environment (functionality demonstrated in near-real world environment; subsystem components fully integrated into system).
Readiness Level 8
Finalized system, process, product, service or tool tested, and shown to operate or function as expected within user’s environment; user training and documentation completed; operator or user approval given.
Readiness Level 9
System, process, product, service or tool deployed and used routinely.
Sources: Readiness Levels – Office of Research Transition and Application (n.d.), NAO 216-115B (n.d.)