Dataset Access

AWS Command Line Interface

One of the most basic ways to access these data is using the AWS Command Line Interface. You can list data in the bucket with the “ls” command:

$ aws s3 ls s3://sentinel-s1-rtc-indigo/

You can use the CLI to browse the bucket and download files to your computer. For example,

$ aws s3 cp s3://sentinel-s1-rtc-indigo/tiles/RTC/1/IW/14/T/PN/2020/S1A_20200801_14TPN_ASC/Gamma0_VV.tif S1A_20200801_14TPN_ASC_Gamma0_VV.tif

will copy the VV image for November 1st, 2019 for tile 14TQK to a local file named S1A_20200801_14TPN_ASC_Gamma0_VV.tif. From here you may proceed to analyze it using software like QGIS, GDAL, R, etc.

GDAL

While you might want to download the files in this dataset locally for analysis, one of the most exciting aspects of this dataset is that it is formatted as Cloud Optimized GeoTIFFs (COGs) which are designed for efficient access from cloud storage services like AWS S3. We can use the support for “virtual file systems” in GDAL to access the files directly off S3. For full details, see the GDAL documentation on the “VSI” virtual file system.

Assuming we have a relatively recent GDAL (>2.1) built with support for cURL, we can read files by specifying them as, /vsis3/[bucket]/[key].

For example, using the example we downloaded in the previous section:

$ gdalinfo /vsis3/sentinel-s1-rtc-indigo/tiles/RTC/1/IW/14/T/PN/2020/S1A_20200801_14TPN_ASC/Gamma0_VV.tif

 Driver: GTiff/GeoTIFF
 Files: /vsis3/sentinel-s1-rtc-indigo/tiles/RTC/1/IW/14/T/PN/2020/S1A_20200801_14TPN_ASC/Gamma0_VV.tif
 Size is 5490, 5490
 Coordinate System is:
 PROJCS["WGS 84 / UTM zone 14N",
     GEOGCS["WGS 84",
         DATUM["WGS_1984",
             SPHEROID["WGS 84",6378137,298.257223563,
                 AUTHORITY["EPSG","7030"]],
             AUTHORITY["EPSG","6326"]],
         PRIMEM["Greenwich",0,
             AUTHORITY["EPSG","8901"]],
         UNIT["degree",0.0174532925199433,
             AUTHORITY["EPSG","9122"]],
         AUTHORITY["EPSG","4326"]],
     PROJECTION["Transverse_Mercator"],
     PARAMETER["latitude_of_origin",0],
     PARAMETER["central_meridian",-99],
     PARAMETER["scale_factor",0.9996],
     PARAMETER["false_easting",500000],
     PARAMETER["false_northing",0],
     UNIT["metre",1,
         AUTHORITY["EPSG","9001"]],
     AXIS["Easting",EAST],
     AXIS["Northing",NORTH],
     AUTHORITY["EPSG","32614"]]
 Data axis to CRS axis mapping: 1,2
 Origin = (600000.000000000000000,4800000.000000000000000)
 Pixel Size = (20.000000000000000,-20.000000000000000)
 Metadata:
   ABSOLUTE_ORBIT_NUMBER=33706
   AREA_OR_POINT=Area
   DATE=2020-08-01
   MISSION_ID=S1A
   NUMBER_SCENES=1
   ORBIT_DIRECTION=ascending
   OVR_RESAMPLING_ALG=AVERAGE
   SCENES=S1A_IW_GRDH_1SDV_20200801T003023_20200801T003048_033706_03E810_267D
   SCENE_1_METADATA={"title": "S1A_IW_GRDH_1SDV_20200801T003023_20200801T003048_033706_03E810_267D", "mission_id": "S1A", "sensor_operational_mode": "IW", "product_type": "RTC", "resolution_class": "H", "processing_level": "1", "polarization_mode": "DV", "start_time": "2020-08-01T00:30:23", "end_time": "2020-08-01T00:30:48", "absolute_orbit_number": "33706", "mission_data_take_id": "03E810", "product_unique_identifier": "267D", "footprint": "{\"type\": \"Polygon\", \"coordinates\": [[[-100.551659, 43.069695], [-97.425003, 43.468075], [-97.102608, 41.97723], [-100.155632, 41.578266], [-100.551659, 43.069695]]]}", "orbit_direction": "ASC", "ingested_at": "2020-08-01T07:50:56.259000", "size": "912783583"}
   SCENE_1_PRODUCT_INFO={"id": "S1A_IW_GRDH_1SDV_20200801T003023_20200801T003048_033706_03E810_267D", "path": "scenes/RTC/1/2020/8/1/IW/DV/S1A_IW_GRDH_1SDV_20200801T003023_20200801T003048_033706_03E810_267D/", "missionId": "S1A", "productType": "RTC", "mode": "IW", "polarization": "DV", "startTime": "2020-08-01T00:30:23", "stopTime": "2020-08-01T00:30:48", "absoluteOrbitNumber": 33706, "missionDataTakeId": "03E810", "productUniqueIdentifier": "267D", "sciHubIngestion": "2020-08-01T07:50:56.259000Z", "s3Ingestion": "2020-09-23T15:05:27.465350Z", "sciHubId": "2a32ac22-1b5f-4222-ad5d-f9f66903f1ed", "footprint": {"type": "Polygon", "coordinates": [[[-100.551659, 43.069695], [-97.425003, 43.468075], [-97.102608, 41.97723], [-100.155632, 41.578266], [-100.551659, 43.069695]]]}}
   TILE_ID=14TPN
   VALID_PIXEL_PERCENT=34.385
 Image Structure Metadata:
   COMPRESSION=DEFLATE
   INTERLEAVE=BAND
 Corner Coordinates:
 Upper Left  (  600000.000, 4800000.000) ( 97d45'58.28"W, 43d20'46.32"N)
 Lower Left  (  600000.000, 4690200.000) ( 97d47' 8.57"W, 42d21'27.51"N)
 Upper Right (  709800.000, 4800000.000) ( 96d24'44.87"W, 43d19'24.88"N)
 Lower Right (  709800.000, 4690200.000) ( 96d27'12.13"W, 42d20' 8.82"N)
 Center      (  654900.000, 4745100.000) ( 97d 6'15.96"W, 42d50'34.05"N)
 Band 1 Block=512x512 Type=Float32, ColorInterp=Gray
   Description = Gamma0_VV
   NoData Value=0
   Overviews: 2745x2745, 1373x1373, 687x687, 344x344

You can also use this format, /vsis3/[bucket]/[key], in any code you write with GDAL to read from COGs off of S3 directly.

Rasterio

Many Python users relate to GDAL through more “Pythonic” wrapper libraries, including rasterio. Using Rasterio to read imagery off S3 is a little simpler than using GDAL because we it handles the translation of s3:// into /vsis3/ for us. We can simply use the same S3 path that we would with the AWS CLI.

For example, we can use the rio info program to inspect the same image we have been using.

$ rio info s3://indigo-sentinel-s1-rtc/tiles/RTC/IW/Gamma0_VV/20191101/14TQK.tif
 {"blockxsize": 512, "blockysize": 512, "bounds": [699960.0, 4390200.0, 809760.0, 4500000.0], "colorinterp": ["gray"], "compress": "deflate", "count": 1, "crs": "EPSG:32614", "descriptions": ["Gamma0_VV"], "driver": "GTiff", "dtype": "float32", "height": 5490, "indexes": [1], "interleave": "band", "lnglat": [-96.00937694546097, 40.11761987007653], "mask_flags": [["nodata"]], "nodata": 0.0, "res": [20.0, 20.0], "shape": [5490, 5490], "tiled": true, "transform": [20.0, 0.0, 699960.0, 0.0, -20.0, 4500000.0, 0.0, 0.0, 1.0], "units": [null], "width": 5490}

The rio command line tool from Rasterio also allows you to specify AWS specific information at the main command group level. For example, this is equivalent:

$ rio info s3://sentinel-s1-rtc-indigo/tiles/RTC/1/IW/14/T/PN/2020/S1A_20200801_14TPN_ASC/Gamma0_VV.tif

You can also use Rasterio to tell GDAL about custom Amazon S3 credentials or configuration settings by passing a rasterio.session.AWSSession object to rasterio.Env:

import rasterio

image = 's3://sentinel-s1-rtc-indigo/tiles/RTC/1/IW/14/T/PN/2020/S1A_20200801_14TPN_ASC/Gamma0_VV.tif'

with rasterio.open(image) as src:
    image_data = src.read()
    # etc etc

To learn more, visit the Rasterio documentation on support for AWS S3. You may also be interested in using or learning from applications that use Rasterio to access COGs, including a number of raster imagery tiling servers:

  • “Marblecutter” dynamic tiler for S3-hosted GeoTIFFs

  • “titiler”, a Rasterio-based tool to help you read webtiles from COGs

  • “Terracotta”, a Python tile server built with Flask, Zappa, & Rasterio