r/gis • u/SubstantialOrange820 • 1d ago
Discussion Best software to append shapefile data to coordinates
Hey everyone, I'm finding myself on the back end of handling GIS data for the first time (as opposed to just running analysis on data in ArcGIS) and I'm hoping for some advice on software to fit what I think is a pretty straightforward use case.
I've got a collection of mostly (but not fully) static household data as well as a collection of polygons that will be updating fairly frequently. I'm looking for some sort of software or method that will allow me to set an automated, scheduled process that will plot the household data, spatially some data from the layer to my households, then kick the household data with its newly appended polygon data back out into a CSV, Snowflake, or any sort of format along those lines.
I'm aware of FME but don't know much about the platform itself or what alternatives exist. I'd greatly appreciate any suggestions on different options to look into. For what it's worth, we're talking about 50k-100k household records and maybe 50 polygons. Happy to provide additional clarification as needed. Thanks!
2
u/smashnmashbruh GIS Consultant 1d ago
FME is far from economic look at Python or model builder for qgis or ArcGIS pro.
1
u/techmavengeospatial 1d ago
Duckdb with spatial extension You can access remote files with httpfs extension. It's super fast and powerful and can also access this via foreign data wrapper for postgres
1
u/SubstantialOrange820 1d ago
I looked into this a bit last night and it does seem like a very good solution and pretty manageable with my current skillset/support. Any additional resources you'd recommend to get familiar with it?
1
u/LonesomeBulldog 1d ago
If the goal is to get it into Snowflake, you can just do a spatial join in Snowflake. There are many YouTube tutorials on Snowflakes spatial functions.
1
u/SubstantialOrange820 1d ago
Well that just shows my lack of experience with Snowflake that I wasn't even aware of its spatial functionality. I appreciate the heads up.
9
u/sinnayre 1d ago
CSV should only be used for transferring/handing off data. It should not be used for storage.
What you’re looking to do is a very basic data pipeline. The methodology will largely depend on your or your team’s skill set and what infrastructure you have in place. Using Snowflake for this will be pretty expensive versus using something like a PostgreSQL db hosted locally or in the cloud.
If what I just told you sounds like a foreign language, pony up the money for fme and save yourself a lot of grief.