RRASSLER Fills A Need by Paying (Geospatial) Technical Debt

Goals:

• To set the background for the state and status of HEC-RAS model data.

Outcomes and Takeaways:

• A deeper understanding of the technical friction experienced working with legacy models and the sorts of questions that are hard to answer without something like RRASSLER.

This presentation

Slide layout: default     Items linked/bordered in green are cited in the tooltip on hover.

: interactive elements     Items linked/bordered in blue are hyperlinked to relevant resources.

There are few hydraulic models as prolific as HEC-RAS, and since it’s first named release in 1995 users have created these models using public and private data and countless hours of engineering scrutinization in order to generate the best possible purpose-built representation of the world. Like any model, some level of input massaging is necessary in order to get the data into the specified mathematical format a model requires. Like most domain specific software solutions, that massaging was rather, forceful, to the point of permanently altering the shape of those inputs into something that most geospatial data readers are unable to handle. This creates a great deal of friction both in terms of model accountability and interoperability, particularly when you take the standpoint as a model consumer. The R based HEC-RAS Wrestler (RRASSLER) is here to mediate that. By internally versioning and aligning data and pointers, the resulting structure provides a bottoms up approach amenable to walking continental scale applications back to the specific point, cross section, and HEC-RAS model they were sourced from.

Background image from: OWP template

Refresher: The HEC-RAS model format

The Theoretical Standard

Files to Track

File grep Pattern (# denotes single numeric wildcard)	HEC-RAS Model Use
.g##	Geometry definitions
.prj	Projection (can be non-standard proj4 string defined file)
.prj	Project (same extension, defines how RAS models are wired)
.p##	Plan file, used to drive the model
.f##	Steady Flow file. Profile information, flow data and boundary conditions
.h##	Hydraulic Design data file
.v##	Velocity file
.o##	Output file
.r##	Run file for steady flow
.u##	unsteady Flow file. Profile information, flow data and boundary conditions
.x##	Run file for unsteady flow
.dss	Data files
.rasmap	Output plan

These files, followed by a .hdf, are transformations of their counterparts needed for newer versions of HEC-RAS and are automatically created as needed.

Sidecar files are fun.

Refresher: The HEC-RAS model format

Model Standards

Files to Track

File grep Pattern (# denotes single numeric wildcard)	HEC-RAS Model Use
.g##	Geometry definitions
.prj	Projection (can be non-standard proj4 string defined file)
.prj	Project (same extension, defines how RAS models are wired)
.p##	Plan file, used to drive the model
.f##	Steady Flow file. Profile information, flow data and boundary conditions
.h##	Hydraulic Design data file
.v##	Velocity file
.o##	Output file
.r##	Run file for steady flow
.u##	unsteady Flow file. Profile information, flow data and boundary conditions
.x##	Run file for unsteady flow
.dss	Data files
.rasmap	Output plan

These files, followed by a .hdf, are transformations of their counterparts needed for newer versions of HEC-RAS and are automatically created as needed.

Sidecar files are fun.

Refresher: The HEC-RAS model format

A “real-world” example

Files to Track

File grep Pattern (# denotes single numeric wildcard)	HEC-RAS Model Use
.g##	Geometry definitions
.prj	Projection (can be non-standard proj4 string defined file)
.prj	Project (same extension, defines how RAS models are wired)
.p##	Plan file, used to drive the model
.f##	Steady Flow file. Profile information, flow data and boundary conditions
.h##	Hydraulic Design data file
.v##	Velocity file
.o##	Output file
.r##	Run file for steady flow
.u##	unsteady Flow file. Profile information, flow data and boundary conditions
.x##	Run file for unsteady flow
.dss	Data files
.rasmap	Output plan

These files, followed by a .hdf, are transformations of their counterparts needed for newer versions of HEC-RAS and are automatically created as needed.

Data from the wild can be a bit unruly.

Even After Overcoming the Pain of Reading in a HEC-RAS model…

Other Potential Readers

• Point and click through the HEC-RAS interface to export
• Con: Unaccountable and unsustainable
• Pro: I can do it
• MCAT-RAS
  • Con: After 2 days of banging my head against the wall, I was not able to get this to run
  • Docker builds with 13/14 failing tests.

Expand for my specific errors

The container builds, and I’m sure the issue is between the keyboard and the chair, but this is very unhelpful and configuration and procedures are very opaque to this simpleton external end user.

user@Brains:/mnt/c/WINDOWS/system32$ cd /mnt/g/Dropbox/root/projects/mcat-ras/Dewberry/mcat-ras/
user@Brains:/mnt/g/Dropbox/root/projects/mcat-ras/Dewberry/mcat-ras$ docker-compose up
[+] Running 2/1
 ✔ Container mcat-ras-api     Created        0.1s
 ✔ Container mcat-ras-newman  Created        0.0s
Attaching to mcat-ras-api, mcat-ras-newman
mcat-ras-api     | /bin/sh: /app/main: not found
mcat-ras-api exited with code 127
mcat-ras-newman  | newman
mcat-ras-newman  |
mcat-ras-newman  | mcat-ras-testing
mcat-ras-newman  |
mcat-ras-newman  |
mcat-ras-newman  | ❏ Forcing Data / Positvies / 2D Unsteady Flow
mcat-ras-newman  | ↳ BaldEagleCrkMulti2D
mcat-ras-newman  |
mcat-ras-newman  |   GET http://api:5600/forcingdata?definition_file=mcat-ras-testing/Example_Projects/2D Unsteady Flow Hydraulics/BaldEagleCrkMulti2D/BaldEagleDamBrk.prj
mcat-ras-newman  | [errored]
mcat-ras-newman  |      getaddrinfo ENOTFOUND api
mcat-ras-newman  |   2. response must be valid and have a body
mcat-ras-newman  |   3. Status code is 200
mcat-ras-newman  |   4. response should have Unsteady
mcat-ras-newman  |   5⠄ JSONError in test-script
mcat-ras-newman  |
mcat-ras-newman  | ❏ Forcing Data / Positvies / General
mcat-ras-newman  |
mcat-ras-newman  | ↳ Bare minimum project
mcat-ras-newman  |
mcat-ras-newman  |   GET http://api:5600/forcingdata?definition_file=mcat-ras-testing/mocked-data/bare-minimum-project/BaldEagleDamBrk.prj
mcat-ras-newman  | [errored]
mcat-ras-newman  |      getaddrinfo ENOTFOUND api
mcat-ras-newman  |   7. response must be valid and have a body
mcat-ras-newman  |   8. Status code is 200
mcat-ras-newman  |   9. response should have Unsteady
mcat-ras-newman  |
mcat-ras-newman  | ↳ No flow files
mcat-ras-newman  |
mcat-ras-newman  |   GET http://api:5600/forcingdata?definition_file=mcat-ras-testing/mocked-data/no-flow-file/BaldEagleDamBrk.prj
mcat-ras-newman  | [errored]
mcat-ras-newman  |      getaddrinfo ENOTFOUND api
mcat-ras-newman  |  11. response must be valid and have a body
mcat-ras-newman  |  12. Status code is 200
mcat-ras-newman  |  13. response should be empty json
mcat-ras-newman  |
mcat-ras-newman  | ❏ Forcing Data / Negatives
mcat-ras-newman  | ↳ File does not exist
mcat-ras-newman  |
mcat-ras-newman  |   GET http://api:5600/forcingdata?definition_file=mcat-ras-testing/Example_Projects/not-exist.prj
mcat-ras-newman  | [errored]
mcat-ras-newman  |      getaddrinfo ENOTFOUND api
mcat-ras-newman  |  15. response must be valid and have a body
mcat-ras-newman  |  16. Status code is equar or greater than 400
mcat-ras-newman  |  17. Bad Reqest
mcat-ras-newman  |
mcat-ras-newman  | ❏ General / Positives
mcat-ras-newman  | ↳ shapefile not a model
mcat-ras-newman  |
mcat-ras-newman  |   GET http://api:5600/isamodel?definition_file=mcat-ras-testing/Example_Projects/2D Unsteady Flow Hydraulics/BaldEagleCrkMulti2D/GISData/MainChannelBanks.prj
mcat-ras-newman  | [errored]
mcat-ras-newman  |      getaddrinfo ENOTFOUND api
mcat-ras-newman  |  19. response must be valid and have a body
mcat-ras-newman  |   ✓  Shapefile not a model
mcat-ras-newman  |
mcat-ras-newman  | ┌─────────────────────────┬──────────┬──────────┐
mcat-ras-newman  | │                         │ executed │   failed │
mcat-ras-newman  | ├─────────────────────────┼──────────┼──────────┤
mcat-ras-newman  | │              iterations │        1 │        0 │
mcat-ras-newman  | ├─────────────────────────┼──────────┼──────────┤
mcat-ras-newman  | │                requests │        5 │        5 │
mcat-ras-newman  | ├─────────────────────────┼──────────┼──────────┤
mcat-ras-newman  | │            test-scripts │       19 │        1 │
mcat-ras-newman  | ├─────────────────────────┼──────────┼──────────┤
mcat-ras-newman  | │      prerequest-scripts │       14 │        0 │
mcat-ras-newman  | ├─────────────────────────┼──────────┼──────────┤
mcat-ras-newman  | │              assertions │       14 │       13 │
mcat-ras-newman  | ├─────────────────────────┴──────────┴──────────┤
mcat-ras-newman  | │ total run duration: 299ms                     │
mcat-ras-newman  | ├───────────────────────────────────────────────┤
mcat-ras-newman  | │ total data received: 0B (approx)              │
mcat-ras-newman  | └───────────────────────────────────────────────┘
mcat-ras-newman  |
mcat-ras-newman  |
mcat-ras-newman  |    #  failure         detail
mcat-ras-newman  |
mcat-ras-newman  |  01.  Error
mcat-ras-newman  |                       getaddrinfo ENOTFOUND api
mcat-ras-newman  |                       at request
mcat-ras-newman  |                       inside ""
mcat-ras-newman  |
mcat-ras-newman  |  02.  AssertionError  response must be valid and have a body
mcat-ras-newman  |                       expected response to have content in body
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / 2D Unsteady Flow / BaldEagleCrkMulti2D"
mcat-ras-newman  |
mcat-ras-newman  |  03.  AssertionError  Status code is 200
mcat-ras-newman  |                       expected PostmanResponse{ …(5) } to have property 'code'
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / 2D Unsteady Flow / BaldEagleCrkMulti2D"
mcat-ras-newman  |
mcat-ras-newman  |  04.  AssertionError  response should have Unsteady
mcat-ras-newman  |                       expected PostmanResponse{ …(5) } to have property 'code'
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / 2D Unsteady Flow / BaldEagleCrkMulti2D"
mcat-ras-newman  |
mcat-ras-newman  |  05.  JSONError
mcat-ras-newman  |                       Unexpected token u in JSON at position 0
mcat-ras-newman  |                       at test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / 2D Unsteady Flow / BaldEagleCrkMulti2D"
mcat-ras-newman  |
mcat-ras-newman  |  06.  Error
mcat-ras-newman  |                       getaddrinfo ENOTFOUND api
mcat-ras-newman  |                       at request
mcat-ras-newman  |                       inside ""
mcat-ras-newman  |
mcat-ras-newman  |  07.  AssertionError  response must be valid and have a body
mcat-ras-newman  |                       expected response to have content in body
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / General / Bare minimum project"
mcat-ras-newman  |
mcat-ras-newman  |  08.  AssertionError  Status code is 200
mcat-ras-newman  |                       expected PostmanResponse{ …(5) } to have property 'code'
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / General / Bare minimum project"
mcat-ras-newman  |
mcat-ras-newman  |  09.  AssertionError  response should have Unsteady
mcat-ras-newman  |                       expected PostmanResponse{ …(5) } to have property 'code'
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / General / Bare minimum project"
mcat-ras-newman  |
mcat-ras-newman  |  10.  Error
mcat-ras-newman  |                       getaddrinfo ENOTFOUND api
mcat-ras-newman  |                       at request
mcat-ras-newman  |                       inside ""
mcat-ras-newman  |
mcat-ras-newman  |  11.  AssertionError  response must be valid and have a body
mcat-ras-newman  |                       expected response to have content in body
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / General / No flow files"
mcat-ras-newman  |
mcat-ras-newman  |  12.  AssertionError  Status code is 200
mcat-ras-newman  |                       expected PostmanResponse{ …(5) } to have property 'code'
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / General / No flow files"
mcat-ras-newman  |
mcat-ras-newman  |  13.  JSONError       response should be empty json
mcat-ras-newman  |                       Unexpected token u in JSON at position 0
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Positvies / General / No flow files"
mcat-ras-newman  |
mcat-ras-newman  |  14.  Error
mcat-ras-newman  |                       getaddrinfo ENOTFOUND api
mcat-ras-newman  |                       at request
mcat-ras-newman  |                       inside ""
mcat-ras-newman  |
mcat-ras-newman  |  15.  AssertionError  response must be valid and have a body
mcat-ras-newman  |                       expected response to have content in body
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Negatives / File does not exist"
mcat-ras-newman  |
mcat-ras-newman  |  16.  AssertionError  Status code is equar or greater than 400
mcat-ras-newman  |                       expected undefined to be a number or a date
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Negatives / File does not exist"
mcat-ras-newman  |
mcat-ras-newman  |  17.  AssertionError  Bad Reqest
mcat-ras-newman  |                       expected response to have status reason 'BAD REQUEST' but got 'UNDEFINED'
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "Forcing Data / Negatives / File does not exist"
mcat-ras-newman  |
mcat-ras-newman  |  18.  Error
mcat-ras-newman  |                       getaddrinfo ENOTFOUND api
mcat-ras-newman  |                       at request
mcat-ras-newman  |                       inside ""
mcat-ras-newman  |
mcat-ras-newman  |  19.  AssertionError  response must be valid and have a body
mcat-ras-newman  |                       expected response to have content in body
mcat-ras-newman  |                       at assertion:0 in test-script
mcat-ras-newman  |                       inside "General / Positives / shapefile not a model"
mcat-ras-newman  |
mcat-ras-newman exited with code 1
user@Brains:/mnt/g/Dropbox/root/projects/mcat-ras/Dewberry/mcat-ras$

• It is unclear from documentation what I should do to resolve this, and only loose guesses as to what the next step would be, assuming this was working for me.
• I am unfamiliar with go, it’s implemented language.
• pyHMT2D is primarily oriented towards 2D.

This pain is not trivial, I weep weekly.

Other archival, tangentially related, under-baked, or circuitous readers. See related utils

• RAS2FIM has readers, which we will extend as part of RRASSLER
• https://github.com/solomonvimal/pyras
• https://github.com/larflows/PyRASFile
• https://github.com/mikebannis/parserasgeo
• https://hatarilabs.com/ih-en/open-modify-run-and-read-output-from-hec-ras-models-with-python-tutorial

Can you answer these questions?

• Where are models?
• What cross sections do I have for my network?
• Is this the same model?

Simple questions, hard answers

Models from Different Providers?

Texas Water Development Board

FEMA Region 6

asynchronous update cycles

• https://www.twdb.texas.gov/flood/science/ble.asp
• https://www.arcgis.com/apps/dashboards/45d3284db7e64694bdcece58c6de39fc

Although the download url is the same, the status of the maps is not the same and makes determining what model an end user is querying for difficult.

Models from Broad Geographies

Clashing spatial units

• Need to pull the entire watershed to get any data
• Take a minute to acknowledge the geographic pain we all just experienced. Am I worried AI will take my job? A boy can dream…
• Helper function

Models at street scale: Example Austin FloodPro

Manual subsetting

• You have to know your street address (easy), or know your watershed name (harder)
• either way are restricted to the watershed scale download
• https://maps.austintexas.gov/FloodPro/

RRASSLER Can!

“Where are models?” – See the mapping vignette!

Thank you for your time - looking forward?

Outcomes and Takeaways:

• A deeper understanding of the technical friction experienced working with legacy models and the sorts of questions that are hard to answer without something like RRASSLER.

Next Steps:

• Build RRASSLER!
• Verify that it’s processing surveys correctly.
• Create a public model catalog.
• Solve Hydrology.

This presentation

See more at the Official RRASSLER documentation.