Release History#

0.21.0 (2023-MM-DD)#

  • BREAKING CHANGES: Rename utils.stack_dict to utils.stack since we are stacking datasets and not dict anymore.

  • BREAKING CHANGES: Band ID for Sentinel-3 OLCI are now int instead of band names (i.e. 7 instead of Oa07. The names don’t change).

  • ENH: Allow to use bands IDs, names and common name added to mapped names when trying to load a spectral band. (#111

  • CI: Update pre-commit hooks

  • DEPS: Remove as many mention as possible to cloudpathlib

  • GITHUB: Update bug template

0.20.4 (2023-09-26)#

  • FIX: Don’t collocate a raster on itself

  • FIX: Better management of default pixel size for multi-resolution products (such as PAN band in Landsat)

  • FIX: Fixing the PAN GSD for Landsat-OLI products

  • FIX: Update some code to match sertit>=1.29.0

0.20.3 (2023-07-31)#

Bug Fixes#

  • FIX: Minor fix in RasterioError handling when reading bands

  • FIX: Fix Landsat L2 surface reflectance and temperature computation (#99)

  • FIX: Fixing TIR temperature conversion for Landsat-7

  • FIX: Test thermal bands in CI

  • FIX: Upgraded to EO STAC extension v1.1.0 (#83)

0.20.2 (2023-06-22)#

Bug Fixes#

  • FIX: Use already computed bands stored in tmp for Planet products

0.20.1 (2023-06-20)#

Bug Fixes#

  • FIX: Allow band aliases (such as "CA" instead of "COASTAL_AEROSOL") in string in band mapping when creating Custom Stacks

Other#

  • COMPAT: Add the alias GREEN_1 for GREEN I band of PlanetScope data, in order to stay in the same pattern as VRE_x, SWIR_x… GREEN1 will be deprecated in another release.

0.20.0 (2023-05-31)#

Breaking Changes#

  • BREAKING CHANGES: Switching from resolution to pixel_size to avoid confusion about the definitions (especially for SAR data) (#82)

  • BREAKING CHANGES: load function now returns a xarray.Dataset (#88)

Bug Fixes#

  • FIX: Collocate bands before trying to create spectral indices: resolve the case where their size mismatches (i.e. in case of window or change of native pixel size)

  • FIX: Landsat band masking when specifying a custom resolution and a custom window

  • FIX: Round the default pixel_size of custom stacks

  • FIX: Convert some fields of STAC items from non JSON serializable dtypes to correct ones

  • FIX: Fix erroneous property set to _get_raw_crs for Maxar products

Other#

  • ENH: Don’t load data into memory when computing indices, ensuring tasks are delayed a bit longer (#58)

  • DOCS: Add documentation about default CRS (#87)

  • DEPS: Dropping support of Python 3.8 (#81)

  • DEPS: Pin sertit to 1.27.0

  • INTERNAL: Better management of logs for deprecation warnings

  • INTERNAL: Refactoring simplify_footprint in sertit library

  • CI: Test that STAC items are serializable when added to a catalog

0.19.4 (2023-04-12)#

Bug Fixes#

  • FIX: Removing calibration step from SNAP pre-processing graph for multi-swath Cosmo-SkyMed 1st GEN products (to avoid ending up with empty images after pre-process)

  • FIX: Fixing the paths to Sentinel-2 quicklooks: using PVI instead of TCI file if no .jpg preview file is found (#84 , #85, thanks a lot @floriandeboissieu)

Other#

  • STAC: Updates in STAC management

  • INTERNAL: Use geopandas.estimate_utm_crs() when possible

0.19.3 (2023-03-24)#

Bug Fixes#

  • OPTIM: Don’t recompute stacks if already existing on disk

  • FIX: Fixing Custom Stacks when specifying datetime=None on creation

  • FIX: Fix regression for multi-swath DGM CSK data (huge region) (#78)

  • FIX: Fix calibration issues with CSK HR data (using fallback GPT graph by default)

Other#

  • OPTIM: Always use chunks when reading rasters (#58)

  • OPTIM: Speed up VRT virtual warping

  • OPTIM: Better management of dask’s usage

  • CI: Fix projection STAC extension’s new version number (1.1.0)

0.19.2 (2023-02-23)#

Bug Fixes#

  • FIX: Fixing stack when saved as integer for some special cases

  • FIX: Clipping negative reflectances to 0 (#79)

  • FIX: Fixing nodata mangement for Theia product

  • FIX: Fixing handling of SCS multi-swath Cosmo-SkyMed products (#78)

  • FIX: Writing spectral indices on disk to align with other bands (#80)

  • FIX: By default, calibration is not applied to slant range CSG data, avoiding producing an empty raster (#48)

Other#

  • OPTIM: Using warped VRT instead of reprojecting DEM/VHR stacks to UTM (#58)

  • TYP: Fixing typos in typing

  • INTERNAL: Moving EOREADER_NAME and DATETIME_FMT into __init__.py

  • INTERNAL: Moving stacking function into utils

  • INTERNAL: Removing unused cache_property decorator

  • INTERNAL: Factorizing _load function

  • CI: Using assert_raster_almost_equal_magnitude in CI to better check according to bands’ content (sertit 1.24.0)

  • DEPS: Officially handling Python 3.11 (adding weekly tests on Python 3.11) (#71)

  • DOC: Updating Custom notebook

  • DOC: Updating jupyter cache to match new way of handling outputs in readthedocs

0.19.1 (2023-01-12)#

Bug Fixes#

  • FIX: Fixing a bug for DIMAP V2 products whith GML masks opening without CRS: assigning first the raw CRS before converting to the product’s CRS

  • FIX: Fixing index creation when exotic bands not handled by ASI have been loaded in the same time (i.e. stacking NDWI with Oa21 band)

Other#

  • CLEAN: Removing useless GCP functions regarding Sentinel-3 data

  • DOC: Adding a Remove Clouds notebook

  • LIB: Pinning sertit to 1.22.0

  • CI: Don’t run tests when only __init__ or __meta__ is updated

  • CI: Some factorizing in gitlab-ci

0.19.0 (2023-01-03)#

Enhancements#

  • ENH: Adding the support of Capella constellation (#74)

  • ENH: Allow the user to load bands with a window (pixels and geo) (#25) , notebook)

Bug Fixes#

  • FIX: Fix extent computation for CSG products with Shapely 2.0

  • FIX: Shapely 2.0 deprecation warnings

Other#

  • DEPR: Add deprecation warning for EOReader spectral indices (used for legacy in 0.18.0) that are aliases of ASI names (#72):

    • AFRI_1_6: AFRI1600,

    • AFRI_2_1: AFRI2100,

    • BSI: BI,

    • NDGRI: NGRDI,

    • NDRE1: NDREI,

    • RGI: RGRI,

    • WV_BI: NHFD,

    • WI: WI2015,

    • RDI: DSI,

    • DSWI: DSWI5,

    • GRI: DSWI4,

    • WV_SI: NDSIWV,

    • PANI: BITM

  • DOC: Changing copyright from 2022 to 2023

0.18.1 (2022-12-08)#

Bug Fixes#

  • FIX: Fix regression for missing EOReader aliases for spyndex spectral indices

  • FIX: Fix bug in footprint computation of DIMAP V1 data

Other#

  • DOC: Add latest DOI link

  • LIB: Pass to sertit==1.21.0 to handle windowed data in read natively

0.18.0 (2022-12-06)#

Breaking Changes#

  • BREAKING CHANGES: Refactoring spectral indices management (#47)

    • Using spyndex library, allowing to use all spectral indices listed here

    • SAR products may now compute indices if possible (see this list)

    • Old EOReader indices are still available for legacy purposes, with some changes:

      • For Sentinel-2 data, the band NIR and NARROW_NIR may be interchanged for some index ( see this discussion)

      • OSAVI formula has changed to stick with the original paper definition (see issue)

      • NDRE2/3 formula are fixed, now using VRE_2/3 and NDRE1 corresponds to NDREI and uses VRE_1

      • CI1 is renamed CI32 and CI2 is renamed CI21 for readability purposes

      • NDWI21 can be written NDWI2100 for homogeneity purposes

      • RDI (or DSI) uses now SWIR_1 instead of SWIR_2 (see this issue)

      • PANI equivalent is now BITM and is normalised ! (divided by 3)

      • SBI is normalized (divided by 2) to fit with BIXS definition

      • ⚠ You may need to install the last spyndex directly from GitHub latest version to have all available indices

  • BREAKING CHANGES: Using pyresample to geocode Sentinel-3 data (#55)

    • Cleaner: better conversion from swath to grid

    • Faster: Up to 4 times faster

    • Allows code refactoring between OLCI and SLSTR

  • BREAKING CHANGES: For SAR product types that are not available in the Data Access Portfolio, default resolution is now the pixel spacing instead of the rg x az resolution

    • Changes mainly Sentinel-1 default resolutions (except from IW mode)

Enhancements#

  • ENH: Adding the support of Harmonized Landsat-Sentinel constellation (#49)

  • ENH: Adding the support of GEOSAT-2 constellation (#59)

Bug Fixes#

  • FIX: Fixing CustomProduct initialization when fields are set to None (instead of not declaring them)

  • FIX: SNAP cannot handle float predictors other than 1! Set it to 1 when saving ortho SAR images to disk, in order for SNAP to be able to despeckle them. See SNAP issue. (#62)

  • FIX: Fixing mix in Sentinel-2 mapping for B8 (NIR, 10m resolution, large spectral bandwidth) and B8A (NARROW_NIR, 20m resolution, narrow spectral bandwidth)

Other#

  • DOC: Add FAQ entry concerning SAR constellations extent KML files failing to be read (TLDR: needs ogr2ogr in your PATH)

  • DOC: Add Technical Note published in Remote Sensing MDPI in Readme

  • DOC: Update optical band mapping graphs (fix regression to 0.15.0 supported constellation)

  • DOC: Add information about DEM management in SAR notebook (#61)

  • DOC: Updating indices paragraphs

  • CI: Using actions/checkout@v3

  • CI: Updating versions of pre-commit hooks

  • LIBS: Updating requirements.txt and setup.py to add pyresample and zarr

0.17.0 (2022-10-12)#

Enhancements#

  • ENH: Adding the support of RapidEye constellation

  • ENH: Handling Planet data with multiple subdatasets (#45)

  • ENH: Adding the support of Landsat Level-2 products (#49)

  • ENH: Adding the support of Pleides Neo SEN and PRJ products (needs GDAL 3.5+ or rasterio 1.3.0+)

  • ENH: Adding the function bands.is_thermal_band

  • ENH: Adding the ability for optical custom stacks to load indices

  • ENH: Adding BAIM (MODIS Burned Area Index) spectral index

  • ENH: Better management of raw units of the bands of optical products

  • ENH: Copying files from tmp_process when changing product’s output

Bug Fixes#

  • FIX: Stacks saved as integers on disk keep their original dtype (float32) in Python

  • FIX: Stacks with bands loaded “as is” are correctly saved as integers on disk (#52)

  • FIX: Using stack CRS (if projected) for DIMAP products instead of recomputing from lat/lon, solving potential discrepancies between stack and product CRS

  • FIX: Workaround for JP2 bug when updating an existing raster (maybe related to this bug)

  • FIX: Better management of SkySat datetime conversion from JSON to XML (deterministic way)

  • FIX: Fixing computation of invalid pixels for Sentinel-2 and DIMAP products (do not remove straylight mask)

  • FIX: Fixing reprojection resolution of VHR data

  • FIX: Computing Brightness Temperature of Landsat TIR bands instead of leaving them as is

  • FIX: Better management of Landsat Instrument values

  • FIX: Better radiometry attribute (adding brightness temperature and reflectance and brightness temperature values)

  • FIX: Changing Brilliance Temperature to the correct Brightness Temperature

  • FIX: Fixing pandas FutureWarning The frame.append method is deprecated and will be removed from pandas in a future version.

  • FIX: Fixing DeprecationWarning invalid escape sequence \.

  • FIX: Manage correctly Planet dubious pixels (especially for 8 bands products)

Optimizations#

  • OPTIM: Reduce memory usage when updating all the bands attributes

  • OPTIM: Reduce memory usage when stacking as integers

Other#

  • DOC: Add the need of using SNAP 8.0 up-to-date or SNAP 9.0 (#42)

  • DOC: Add the STAC session in API documentation

  • DOC: Add warnings for shifts when orthorectifying DIMAP SEN products (using RPCs) (#53)

  • DOC: Add limitations to custom stacks

  • DEPS: Dropping support of Python 3.7 (#18)

  • DEPS: Update minimum version of libs (geopandas 0.11.0+, rasterio 1.3.0+…)

0.16.1 (2022-08-03)#

Bug Fixes#

  • FIX: Add the missing conversion to reflectance for Sentinel-3 OLCI

  • FIX: Better condition for the conversion to reflectance for Sentinel-2 THEIA

  • FIX: Add logs for SkySat data that cannot been converted to reflectance and fix the radiometry field of its band xarrays

  • FIX: Add the correct nodata (when overridden by the user) to stacks saved as uint16

Optimizations#

  • OPTIM: Reduce memory usage during stacking

Other#

  • CI: Test reflectance values

0.16.0 (2022-08-01)#

Enhancements#

  • ENH: Adding the support of SuperView-1 constellation (#21)

  • ENH: Adding the support of SPOT-4/5 constellations (#39)

  • ENH: Allow the possibility to pass a constellation (or a constellation list) to Reader().open() to speed up the opening of a product

  • ENH: Add a quicklook search for Sentinel-3 products

Bug Fixes#

  • FIX: Fix quicklook media type with JP2 files

  • FIX: Fix Sentinel-3 SLSTR F1 bands based on F grid

  • FIX: Correct the UTM projection for Sentinel-3 data

  • FIX: Fix handling of zipped Sentinel-2 L2Ap

  • FIX: Fix zipped Sentinel-2 with other XML files in GRANULE subdirectories

Other#

  • Renaming master branch to main

0.15.1 (2022-06-02)#

Optimizations#

  • OPTIM: Try to create Vision-1 footprint from the preview file instead of from the stack.

  • OPTIM: Create footprints for stacked products (i.e. Maxar, SkySat, Custom…) without mask by opening only the first band of the stack

  • OPTIM: Create footprints for Maxar Products with a resolution 10 times lower.

  • OPTIM: Footprints have now maximum 50 vertices in order to avoid pixelized footprints

Bug Fixes#

  • FIX: Fixing condensed name to avoid duplicates:

    • adding the job_id for VHR products

    • adding the polarization channels for SAR products

  • FIX: Remove import of pystac in stac_utils

  • FIX: Fix bug for Vision-1 data looking for non-existing RPC files in case of ORTP product type

  • FIX: Fix quicklook regex for Vision-1 data

  • FIX: Fix regex for raw bands for extracted Sentinel-3 OLCI products

  • FIX: Fix PlanetScope identifying regex to handle products with a satellite_id containing a letter

  • FIX: Force metadata regex for Maxar products to look for a file with pattern {name}.XML to avoid other misplaced XML to be found in place of the true XML.

  • FIX: Fix regression for Landsat-7 footprint

  • FIX: Manage the case with cloud_cover = -999.0 for Maxar products (returns None)

Other#

  • CI: Add new optical products to be tested for end-to-end tests

0.15.0 (2022-05-30)#

Breaking Changes#

  • BREAKING CHANGES: Optical becomes Spectral when more appropriate

  • BREAKING CHANGES: Platform and Sensor become Constellation when more appropriate, to fit STAC vocabulary (#29):

    • Platform enum becomes Constellation

    • prod.platform becomes prod.constellation

    • prod.sat_id becomes prod.constellation_id

  • BREAKING CHANGES: File alias is removed, replaced by *_bands files and proper imports in bands.__init__

  • BREAKING CHANGES: Product attribute band_names becomes bands in order to be STAC compliant (#29)

  • BREAKING CHANGES: Better use of NIR and NARROW_NIR in the indices file (according to the gsd of Sentinel-2 bands composing the indices)

  • BREAKING CHANGES: Correcting Landsat product types to better manage processing levels and instrument. Landsat-8/9 condensed name may change!

Enhancements#

  • ENH: Adding the support of SkySat (Collect) products (#20)

  • ENH: Bands in mapping are now objects, instead of just IDs (#29). This allows us to:

    • Add band metadata (such as center wavelength, bandwidth…)

    • Map spectral bands between STAC spec and EOReader format (#29)

    • Add a better __repr__ functions

  • ENH: Handling 8 bands PlanetScope data (#20)

  • ENH: Adding the GREEN1 mapped band, corresponding to PlanetScope GREEN I and Sentinel-3 OLCI Oa05 band

  • ENH: Handle some slightly broken Sentinel-2 products:

    • when the metadata files are corrupted or when the detfoo vectors are empty (#34)

    • with missing MSK prefix for QI_DATA files (i.e. DETFOO instead of MSK_DETFOO)

  • ENH: Handle exception for corrupted bands (in Sentinel-2 and utils.read) (#34)

  • ENH: Add a STAC object that can be used to retrieve STAC Items from every Product (prod.stac.create_item()) (#29)

  • ENH: Add a get_mean_viewing_angles for Optical Products to fill STAC View Extension (#29)

  • ENH: Extending get_raw_band_paths to every product (#31)

  • ENH: Adding a is_ortho attribute corresponding to when the product is already orthorectified/geocoded, in order to avoid computing heavy processes without wanting it (i.e. footprint…)

  • ENH: Adding the instrument name of every constellation, under prod.instrument

  • ENH: Handling COSMO product with only the h5 file in it (if missing XML metadata file) (#36)

Optimizations#

  • OPTIM: Retrieve name from filename if possible

  • OPTIM: Retrieve extent from metadata when possible (for VHR data)

Bug Fixes#

  • FIX: Fixing the band mapping of WorldView-2/3 Multi (8 bands)

  • FIX: Retrieval (if possible) of Sentinel-1 unique ID (was missing from the product name, as it is not in the product preview)

  • FIX: Fixing PAZ/TDX MTD regex

  • FIX: Optical products: Only set cloud cover and radiometry attributes if spectral bands are asked

Other#

  • INTERNAL: File spot_6 and spot_7 are removed, replaced by a unique spot file. This shouldn’t affect the user.

  • INTERNAL: Refactoring Landsat-XX products into LandsatProduct, this should be invisible for user.

  • INTERNAL: Some refactoring in VHR files

  • WARNINGS: Filter warnings from __init__

  • CI: Do not process two times the zipped Sentinel-1 in end-to-end tests and manage when the runner kills SNAP

  • DOC: Adding a STAC notebook

  • DOC: Various updates

0.14.0 (2022-04-14)#

Breaking Changes#

  • BREAKING CHANGES: footprint, extent, wgs84_extent and crs properties are converted back to methods in order to prevent side effects of expensive computation when displaying the object when debugging (rollback before version 0.8.0)

  • BREAKING CHANGES: get_all_index becomes get_all_indices

  • BREAKING CHANGES: acquisition_datetime becomes datetime and default_resolutionbecomes resolution for CustomProduct in accepted keywords, and the metadata fields have been renamed according to the CustomFields enum

Enhancements#

  • ENH: Adding spectral indices:

    • Shadow Index (SI)

    • Global Vegetation Moisture Index (GVMI)

    • Soil Brightness Index (SBI), Soil Cuirass Index (SCI)

    • Panchromatic mocking Index (PANI)

    • Green-to-Red ratio Index (GRI)

    • Soil Adjusted Vegetation Index (SAVI)

    • Optimized Soil Adjusted Vegetation Index (OSAVI)

    • Visible Atmospherically Resistant Index (Green) (VARI)

    • Enhanced Vegetation Index (EVI)

    • Chlorophyll Index RedEdge VRE_3/VRE_2 (CI1)

    • Chlorophyll Index RedEdge VRE_2/VRE_1 (CI2)

    • Normalized Difference Moisture Index (with SWIR_21) (NDMI21)

  • ENH: Making SAR attribute snap_filename public

  • ENH: Handling ICEYE pure SLC products

  • ENH: Allowing the user to choose if they want the GRD or SLC image for ICEYE products

  • ENH: Add the possibility to directly load the cloud cover for optical data (and add it in the band attributes) (#28)

  • ENH: Add the possibility to retrieve the quicklook path (if existing) and add the plot function allowing the user to plot the quicklook (if existing) (#28)

  • ENH: Add the possibility to retrieve the orbit direction (and add it in the band attributes) (#28)

Bug Fixes#

  • FIX: Fixing the inversion between 8 and 8A bands for Sentinel-2 and Sentinel-2 Theia products

  • FIX: Loading every optical band in reflectance (fixed for Sentinel-2 THEIA, Maxar, Planet and Vision-1 data) (#30)

  • FIX: Fixing ReferenceError: weakly-referenced object no longer exists when deleting an object

  • FIX: Do not set sea values to nodata when orthorectifying SAR data with SNAP

  • FIX: Handle Sentinel-2 data with PB < 02.07 as L2Ap products

  • FIX: Fixing nodata and offset for Sentinel-2 data with PB > 04.00

  • FIX: Handle new ICEYE metadata name’s nomenclature

  • FIX: Fixing harmless regex error when searching for B1 path for Landsat products

  • FIX: Fixing platform for Sentinel-2 Theia

Other#

  • DOC: Creating a real base notebook and renaming the old one to optical

  • DOC: Better type hints (replacing XDS_TYPE by xr.DataArray)

  • CI: Using sertit.ci.reduce_verbosity instead of recreating the function

0.13.1 (2022-03-08)#

Bug Fixes#

  • FIX: Handling Sentinel-2 L2Ap data

  • FIX: Do not use --no-binary fiona,rasterio directly in requirements.txt (breaks on Windows)

  • FIX: Fixing stacking with string bands

  • FIX: Better __repr__ function

  • FIX: Read README as UTF-8 in setup.py

Other#

  • CI: Adding a tag for choosing the runners

  • DOC: Fixing cartopy/GEOS conflicts making the documentation build to fail

0.13.0 (2022-03-02)#

Enhancements#

  • ENH: Adding the support of Landsat-9 sensor

  • ENH: Support Sentinel-2 with missing datatake metadata file(sometimes happens with data downloaded from AWS buckets and converted to .SAFE)

Bug Fixes#

  • FIX: Using default SAR resolution from official Copernicus Data Access Portfolio (2014-2022) ( Sentinel-2 default resolution goes to 10.0 m !)

  • FIX: Use --no-binary fiona,rasterio directly in requirements.txt

  • FIX: Removing useless outputComplex line in GPT graphs that is breaking SNAP on Linux

  • FIX: Removing the workarounds caused by some bugs of cloudpathlib and enabling retrieval of nested SAR products (TSX, TDX, PAZ, RCM) from S3 compatible storage.

  • FIX: Do not process nodata for a band already existing

  • FIX: Fixing an error when reading TIR bands with Landsat-7

  • FIX: Fixing an error when additive/multiplicative coefficients are set to NULL for Landsat data

  • FIX: Returning sun angles always as float (some Sentinel-3 angles were returned as np.array)

Other#

  • CI: Do not try to process SAR end to end if GPT cannot be found

  • CI: Publishing wheel from GitHub instead of Gitlab

  • REPO: Setting GitHub as the main repository and using new Gitlab runners

0.12.0 (2022-02-09)#

Enhancements#

  • ENH: Adding the support of Pleiades-Neo, Vision-1 and SAOCOM sensors

  • ENH: Adding a keyword to allow passing a specific DEM path in load/stack (for VHR orthorectification and DEM bands)

  • ENH: Adding the name of the DEM in DEM band (i.e. allow to compute the HILLSHADE with a DEM and the SLOPE with a DTM)

Bug Fixes#

  • FIX: Sentinel-2 Processing Baseline 04.00: NARROW_NIR bands are now loaded correctly

  • FIX: Maxar products (with Multi band ID) are now correctly handled

  • FIX: Using COPDEM-30 (GLO-30) by default for SNAP as it appears that the retrieval has been fixed.

  • FIX: Fixing the default name for cleaned bands for Sentinel-3 SLSTR data (was set on CLEAN instead of NODATA)

  • FIX: Fixing default band for Custom stacks

  • FIX: Fixing get_existing_band_paths behavior for Custom stacks

  • FIX: Remove other never covered lines of code (archived RCM products, complex ICEYE products, others…)

  • FIX: Re-enabling loading str bands (regression)

  • FIX: Proper check for empty fields when parsing metadata

  • FIX: VHR _get_dem_path raises ValueError instead of TypeError

  • FIX: Pre-process SAR bands before despeckling if not existing (was OK in most of the cases, but broke in some cases, especially with CI folder activated and S3 compatible storage)

  • FIX: Remove warning invalid escape sequence \., \w, \D and \s

  • FIX: Do not set long_name for RAW_CLOUDS arrays

  • FIX: Providing a URL DEM on Windows throws a OSError instead of a bare Exception

Optimizations#

  • OPTIM: Do not pre-process existing Sentinel-3 geocoded bands

  • OPTIM: Do not look for valid metadata further than a given nested level in product’s directory (for extracted products)

Other#

  • CI: Using another (faster) runner

  • CI: Add on disk and end-to-end tests

  • CI: Do not write tmp files when running on disk tests

  • CI: Coverage:

    • Get coverage as HTML

    • Remove useless lines from coverage

    • Combine coverage of S3 and on disk tests

  • DOC: Adding a DEM notebook

0.11.2 (2022-01-19)#

Bug Fixes#

  • FIX: Fixing archived SAR processing

  • FIX: Needs extraction for RS2-SLC data as SNAP does not handle the product

  • FIX: Fixing the default name for cleaned bands for optical data (was set on CLEAN instead of NODATA)

0.11.1 (2022-01-17)#

Bug Fixes#

  • FIX: Fixing complex and orthorectified products for SAR data

  • FIX: Fixing RADARSAT-2 SLC product type

Optimizations#

  • OPTIM: Only preprocessing wanted SAR bands (instead of all existing)

  • OPTIM: Do not interpolate nan values by default when writing SAR bands to disk (using a keyword instead)

Other#

  • DOC: Updating the SAR notebook and documentation

0.11.0 (2022-01-13)#

Breaking Changes#

  • BREAKING CHANGES: Renamed is_band to is_sat_band to better reflect that this function only checks optical and SAR bands

  • BREAKING CHANGES: Invalid pixels are not processed by default anymore! Only the nodata is set (to go a bit faster)

Enhancements#

  • ENH: Allowing the user to choose the pixel processing for optical bands: raw band, only nodata or total cleaning of defective pixels (#16)

  • ENH: Adding a CustomProduct, allowing the user to load any stack as an EOReader Product !

  • ENH: Check if a band exists before trying to load it

Bug Fixes#

  • FIX: Better handling of __all__ in __init__.py files

  • FIX: Ensure that extents and footprints are in UTM

  • FIX: Removing docs from wheel

  • FIX: Fixing TIR bands reading for Landsat data

Optimizations#

  • OPTIM: Optimizing manage_invalid_pixels for Sentinel-2 data (processing baseline >= 04.00)

Other#

  • DOC: Update README, documentation and notebooks

  • DOC: Water Extraction notebook has been refined to show how to manage multiple products

  • DOC: Update the installation paragraph in README

  • DOC: Adding a For Contributors section in the documentation (contributing, release history and GitHub repository)

  • DOC: Remove doc testing in GitHub (as the docs are built with readthedocs)

  • INTERNAL: Better management of project metadata (version…) in a dedicated file

0.10.1 (2022-01-04)#

Bug Fixes#

  • FIX: Resolve a bug when methodtools is not present (for conda package)

0.10.0 (2022-01-04)#

Enhancements#

  • ENH: Adding has_bands to products, ingesting lists as a shortcut for testing the availability of multiple bands

  • ENH: Simplifying imports. Now you can replace:

    • from eoreader.bands.alias import RED, NDVI by from eoreader.bands import RED, NDVI,

    • from eoreader.products.optical.optical_product import OpticalProduct by from eoreader.products import OpticalProduct,

    • from eoreader.products.optical.s3_slstr_product import SlstrRadAdjustTuple by from eoreader.products import SlstrRadAdjustTuple, …

Optimizations#

  • OPTIM: Writing cloud bands on disk to speed up multiple calls to load or stack functions (#17)

Bug Fixes#

  • FIX: Correctly naming cloud xarrays

  • FIX: Add missing SLEA (Spot Extended Area) product type to ICEYE data

  • FIX: Sentinel-2 clouds (with processing baseline >= 4.0) are now given with a rasterio shape (count, height, width)

Other#

  • CI: Remove pages stage and run only the tests when a Python file has changed

  • DOC: Updating notebooks

  • DOC: Updating copyright to 2022

0.9.5 (2021-12-14)#

Bug Fixes#

  • FIX: Do not force import methodtools (not existing lib in conda)

  • FIX: Using GRD resolution given by the constructors as default values for SLC products. Do not look it up in metadata as SLC resolution is NOT the GRD resolution !

0.9.4 (2021-12-13)#

Bug Fixes#

  • FIX: Caching properties and functions only for object instances

  • FIX: Fixing metadata reading for COSMO-SkyMed 1st Generation with Wide Region and complex product type (handling of multiple swaths)

  • FIX: Updates of SNAP GPT graphs for complex SAR data

  • FIX: Interpolate nodata inside SAR images (badly handled by SNAP -> fill the gaps that shouldn’t exist)

Other#

  • INTERNAL: Creation of a class CosmoProduct handling generic methods for both COSMO-SkyMed generations

0.9.3 (2021-12-09)#

Bug Fixes#

  • FIX: Fixing the search for .TIL files for Maxar products (with on disk files)

  • FIX: Fixing the search for metadata files for Landsat products (with on disk files)

  • FIX: Fixing the search for metadata files for TerraSAR-X, TanDEM-X and PAZ SAR products (with on disk files)

  • FIX: Fixing SNAP files for TerraSAR-X, TanDEM-X and PAZ SAR products

  • FIX: Fixing when reading CRS code for DIMAP products

0.9.2 (2021-12-07)#

Bug Fixes#

  • FIX: Fixing flag type for Sentinel-3 data

  • FIX: Do not multiply the flags values by the radiance adjustment factor for Sentinel-3 SLSTR!

  • FIX: Fixing flag exception threshold for Sentinel-3 SLSTR

  • FIX: Fixing preprocessed band filenames for Sentinel-3 SLSTR

0.9.1 (2021-12-07)#

Bug Fixes#

  • FIX: Reader().valid_mtd now correctly accepts strings instead of only Platform objects

  • FIX: Better handling of Sentinel-2 product type

  • FIX: Save bands’ new attributes in str (to pickle them)

  • FIX: Add a clear() function to clear products cache

0.9.0 (2021-12-06)#

Enhancements#

  • ENH: Adding the support of the ICEYE sensor

  • ENH: Adding the support of the COSMO-SkyMed 2nd Generation sensor

  • ENH: Adding some attributes to bands and stack: sensor, sensor_id, product_type, acquisition_date , condensed_name #7

  • ENH: Replace name by filename and read directly the true name of the product in the metadata (#15)

Bug Fixes#

  • FIX: Sentinel-1 metadata file with archived products (discarding RFI folder in its search).

  • FIX: Add Quickbird, GeoEye and WorldView sensors in reader regexes.

  • FIX: Add scipy in requirements.txt and setup.py

Other#

  • DOC: Fix references to pcigeomatics that doesn’t exist anymore (RADARSAT-2 and Constellation)

  • REQ: Update dask to fix a security issue (only in requirements as dask is not mandatory)

0.8.1 (2021-10-26)#

Bug Fixes#

  • FIX: Do not force chunk in utils.read if dask is not installed

0.8.0 (2021-10-25)#

Breaking Changes#

  • BREAKING CHANGE: crs, footprint, extent, wgs84_extent are now properties !

  • BREAKING CHANGE: Removing raw gdaldem CLI from EOReader (the HILLSHADE and SLOPE bands are now slightly different !) (#10)

  • BREAKING CHANGE: HILLSHADE is given in float32 instead of uint8

  • BREAKING CHANGE: SLOPE is given in degrees instead of percents

Enhancements#

  • ENH: Adding the support of the PAZ SAR sensor

  • ENH: Adding the support of the Sentinel-2 processed with the processing baseline 4.0 (#11)

  • ENH: Removing SNAP from Sentinel-3 pre-process -> Freeing optical data from SNAP dependency ! (#12)

  • ENH: Enabling the use of other S3-SLSTR suffixes than an (stripe A at nadir position)

  • ENH: Thermal bands of Sentinel-3 SLSTR can now be used

  • ENH: All bands of Sentinel-3 SLSTR/OLCI can now be used (S7, F1, F2 for SLSTR, Oaxx for OLCI) (#14)

  • ENH: YELLOW band is mapped to Oa07 band of Sentinel-3 OLCI

  • ENH: Zipped Sentinel-3 products can now be processed

  • ENH: Allow the use of kwargs in load, mainly for rasters.read (and allowing i.e. radiance adjustment in S3-SLSTR)

Optimizations#

  • OPTIM: crs, footprint, extent, default_transform, wgs84_extent are cached ( using @cached_property) (#13)

  • OPTIM: get_mean_sun_angles and default_transform are now cached ( using @cache) (#13)

  • OPTIM: get_datetime: Look for the date only if datetime attribute is None (#13)

  • OPTIM: Better management of fspath for cloud-stored products (download the files only once)

  • OPTIM: Stop downloading/extracting files if not necessary

Bug Fixes#

  • FIX: Bands are correctly ordered in stacks

  • FIX: Only load a band once, even if asked several time in the bands

  • FIX: Use band size for cleaning optical pixel (instead of user resolution/size)

  • FIX: Always take the absolute value of the resolution when converting it to strings (for filenames)

  • FIX: Take the default resolution if nothing is given when converting it to strings (for filenames)

  • FIX: Always use utils.read/write instead of rasters.read/write (for Dask management)

  • FIX: Fixing a bug in utils.write

  • FIX: Add .xml files from eoreader/data in the MANIFEST.in

  • FIX: Add forgotten @abstractmethod where needed

  • FIX: Better management of _tmp_process

  • FIX: Fixing minor bug when trying to read metadata with a POSIX path

  • FIX: Fixing the **kwargs omission in utils.read

  • FIX: Better management of _temp_process directory

  • FIX: Landsats and TSX: Can use other filenames now

Other#

  • DEPR: FAR_NIR band is removed

  • REQ: Using h5netcdf instead of netCDF4

  • DOC: Add a Context paragraph in the README

  • DOC: Add a Conda x SNAP question in the FAQ

  • DOC: Creation of a Sentinel-3 notebook

  • DOC: Updates of notebooks

  • DOC: Numerous updates

0.7.1 (2021-09-29)#

Bug Fixes#

  • FIX: Fixing a bug when opening archived Sentinel-1 (wrong metadata file found)

  • DOC: Updating CSS and readme

0.7.0 (2021-09-23)#

Enhancements#

  • ENH: Implementing RADARSAT-Constellation products (as RCM)

  • ENH: Implementing Maxar products (such as GE01, WV02, WV03, WV04, but others should be supported too)

  • ENH: Implementing TanDEM-X products (as TDX)

  • ENH: Adding RH, RV, RH_DSPK and RV_DSPK SAR bands

  • ENH: Adding the YELLOW optical band (for WorldView-2, WorldView-3 and Sentinel-3 OLCI)

  • ENH: Adding WorldView index (without the ones using SWIR)

  • ENH: Loading by size -> round resolution to the closest meter (or decimeter for resolution < 1.0m)

  • ENH: Super class for VHR data

Bug Fixes#

  • FIX: Fixing reading PlanetScope archived products (error in read band)

  • FIX: Fix band name with complex resolutions

  • FIX: Fixing minor bug in RADARSAT-2 data when looking for product type

  • FIX: Fixing SAR band search in BEAM-DIMAP files

  • FIX: Fixing python version in environment.yml

  • FIX: Discard unused MIR and FNIR bands

  • FIX: Check for existence of given path when reading any product

  • FIX: Workaround for a bug involving some downloaded but badly formatted archives for Sentinel-2

  • FIX: Allow NARROW_NIR for and DIMAP data (== NIR)

  • FIX: Better management of writeable band folder

Other#

  • DOC: Fix documentation of the NDWI index

  • DOC: Update graph for optical band mapping

  • CI: Adding a test loading invalid band name

  • CI: Setting CI log level to DEBUG

  • CI: Accelerating the CI processes

0.6.4 (2021-09-15)#

Bug Fixes#

  • FIX: Sentinel-3 band mapping (Coastal Aerosol <-> 03, BLUE <-> 04)

Other#

  • DOC: Adding an interactive graph for optical band mapping

0.6.3 (2021-09-10)#

Bug Fixes#

  • FIX: Load works with string bands (prod.load('BLUE'))

  • FIX: Fixing missing _remove_tmp_process for products needing extraction

  • FIX: Remove multi converting for Sentinel-3

0.6.2 (2021-09-10)#

Bug Fixes#

  • FIX: Better handling of archives for products that needs extraction

  • FIX: TerraSAR-X products need to be extracted to be processed by SNAP !

0.6.1 (2021-09-10)#

Bug Fixes#

  • FIX: Fixing critical bug for Sentinel-3 (mapping between clean bands and SNAP bands)

0.6.0 (2021-09-02)#

Enhancements#

  • ENH: Ensuring EOReader supports Dask

Bug Fixes#

  • FIX: Fixing and adding BAIS2 index in alias

  • FIX: Fixing GLI index

  • FIX: Fixing a bug when writing reprojected DIMAP band

  • FIX: Fixing a bug with SCS Cosmo-SkyMed data

Other#

  • DOC: Adding a DASK notebook

  • DOC: Updating notebooks

0.5.0 (2021-08-24)#

Breaking Changes#

  • BREAKING CHANGE: Read metadata/namespaces only once and store it as a private member. Keep accessing it through the read_mtd function (#9) WARNING: Breaking change for Landsat: read_mtd() loses the argument force_pd=True as it always returns an Etree

Enhancements#

  • ENH: Adding the BAIS2 index

  • ENH: Reads Sentinel-3 global attributes as metadata:

    • absolute_orbit_number

    • comment

    • contact

    • creation_time

    • history

    • institution

    • netCDF_version

    • product_name

    • references

    • resolution

    • source

    • start_offset

    • start_time

    • stop_time

    • title

    • ac_subsampling_factor (OLCI only)

    • al_subsampling_factor (OLCI only)

    • track_offset (SLSTR only)

  • ENH: Refining Despeckle Graph (#6) to use a more usual filter (Refined Lee)

  • ENH: Allowing the user to open the datatake metadata for Sentinel-2 products

Bug Fixes#

  • FIX: Decoupling classic metadata reading from the name as EOReader accepts now modified product names (#9)

  • FIX: Better handling of cloud-stored DEM (raising an exception for non-ortho DIMAP data as GDAL and rasterio does not handle that case)

  • FIX: environment.yml to respect the stricter use of file: syntax. See here for more information.

  • FIX: Fixing bug when opening an archive product with name mode and nested dictionary (when looking for a filename instead of the directory name)

Other#

  • CI: Fixing test_dems_https and resetting DEM afterward

  • CI: Fixing DEM and ds2 database management

  • DOC: Adding a FAQ page and enhancing the Main Features page (#3)

  • DOC: Read Metadata has its own paragraph in Main Features

0.4.8 (2021-07-23)#

Enhancements#

  • ENH: Allowing stack to take single band in input instead of a list

Bug Fixes#

  • FIX: Fixing a regression loading optical bands which have been previously cleaned (Landsat, Theia, possibly PlanetScope)

  • FIX: load and stack always returns float32 arrays

Other#

  • CI: Testing if loading 2 times a band gives the same result

0.4.7.post0 (2021-07-23)#

Bug Fixes#

  • FIX: Fixing a regression loading Landsat bands which have been previously cleaned

0.4.7 (2021-07-23)#

Enhancements#

  • ENH: Adding a default_transform function returning data from default band (without warping it) -> mapping calculate_default_transform from rasterio

  • ENH: Adding a clean_tmp function allowing the user to clean the product’s temporary output by hand

  • ENH: Simplifying DEM warping code

Bug Fixes#

  • FIX: DIMAP products return always projected (in UTM) default bands (get_default_band_path uses _get_default_utm_band)

  • FIX: Theia Footprint returns a GeoDataFrame instead of a GeoSeries

  • FIX: Better management of the size keyword with load and stack functions

  • FIX: Landsat retrieval of multipart cleaned bands (like SWIR_1)

  • FIX: Some typehints fixes

  • FIX: Sentinel-3 open sun angles with NetCDF files stored in the cloud

  • FIX: Silently pass the writing of clean bands if we cannot (like if permission error)

Other#

  • CI: Do not reinstall everything if not needed (only PyYAML)

  • CI: renaming build stage into lint

  • CI: simplifying geometry comparison

  • CI: Testing the size keyword for every sensor

0.4.6 (2021-07-19)#

Bug Fixes#

  • FIX: Fixing no data for Sentinel-3 cloud bands

  • FIX: In alias: DeprecationWarning: using non-Enums in containment checks will raise TypeError in Python 3.8

Other#

  • CI: Set default S3 client to point to unistra’s bucket

  • CI: Tox SNAP relay S3_DB_URL_ROOT

0.4.5 (2021-07-13)#

Bug Fixes#

  • FIX: Adding condensed name in the search when loading S3-SLSTR clouds

  • FIX: Fixing no data for Sentinel-3 bands processed by SNAP

  • FIX: Fix bug when stack path’s directory doesn’t exist

0.4.4 (2021-07-13)#

Bug Fixes#

  • FIX: Do not verbose empty lists when loading optical bands

  • FIX: Sentinel-3: Bands processed by SNAP are written with the condensed name as suffix

0.4.3.post1 (2021-07-08)#

Bug Fixes#

  • FIX: Fixing another bug for DEM_PATH using S3 Paths

Other#

  • DOC: Add a DOI

0.4.3.post0 (2021-07-08)#

Bug Fixes#

  • FIX: Fixing DEM_PATH using S3 Paths

0.4.3 (2021-07-05)#

Enhancements#

  • ENH: Optimizing loading cloud bands for DIMAP Products

  • stack accepts **kwargs in order to pass options to rioxarray.to_raster()

Bug Fixes#

  • FIX: Fixing not found masks with S3+zip Sentinel-2 products

  • FIX: Fixing some type hints

Other#

  • CI: Fixing network directories with pathlib

0.4.2 (2021-07-01)#

Enhancements#

  • ENH: Enabling the use of products stored in the cloud (S3, S3 compatible storage, Google, Azure…) through cloudpathlib

  • ENH: Using correct band names in long_name

Other#

  • CI: Use pre-computed cleaned band if existing

  • DOC: Adding examples for using S3 data, especially for S3 compatible storage

0.4.1.post0 (2021-06-21)#

Bug Fixes#

  • FIX: cloud mask values were inverted in Sentinel-2 cloud masks

  • FIX: Landsat collection 2 cloud masks are now OK

0.4.1 (2021-06-21)#

Bug Fixes#

  • FIX: Improving stacks saved as uint16:

    • Only satellite bands and index are scaled (*10.000)

    • DEM bands are just rounded

    • Cloud bands (booleans) are saved as is

  • FIX: Fixing a rasterization bug affecting S2 and DIMAP masks, happening when the vectors have another size than the image

  • FIX: Adding a warning on bad georeferencing when using GS and GT Landsat products

Other#

  • CI: Minor updates in documentation and code

0.4.0 (2021-06-10)#

Enhancements#

  • ENH: Adding THR data support:

    • PlanetScope

    • Pleiades

    • SPOT-6/7

  • ENH: Better handling of SNAP DEMs (using External DEM and other available SNAP DEMs)

Bug Fixes#

  • FIX: More robust way of looking for data directory

  • FIX: Bug fix in stack that causes some bands to be inexplicably empty sometimes

  • FIX: Bug fix in alias.isindex

  • FIX: Forcing extent to UTM

Optimizations#

  • OPTIM: Write clean bands on disk to avoid redoing invalid pixel computation and allow the user to remove them on deletion

  • OPTIM: prod.has_bands / prod.get_existing_bands do not orthorectify/despeckle SAR bands anymore

Other#

  • CI: Adding a bimonthly test for SNAP processes

  • DOC: Adding two new notebooks (SAR and VHR data)

  • DOC: Completing the documentation

0.3.4 (2021-05-28)#

  • BREAKING CHANGES: read_mtd() returns a dict for the namespace map in order to manage multi namespace XMLs

  • ENH: Introduced support for DEM files from web urls (starting with http(s)://)

  • FIX: Landsat Zenith angle computation (mixing elevation and zenith angle)

  • CI: Adding weekly tests (tox on Python 3.7, 3.8, 3.9 on Linux and Windows)

  • DOC: Updating documentation, setting DEM path as an environment variable

0.3.3 (2021-05-21)#

  • Migrating documentation to sphinx and readthedocs

  • Refactoring documentation and ReadMe

  • Bug correction when loading invalid masks for S2 sensor (fiona.errors.UnsupportedGeometryTypeError)

0.3.2.post2 (2021-05-04)#

  • Bug when reading cloud mask on Linux

  • Log update in product._check_dem_path()

  • README & documentation updates

0.3.2.post1 (2021-05-04)#

0.3.2 (2021-05-04)#

  • In case of DEM bands, checks if the DEM is set and exists and raise an exception if not

  • Setting minimum versions in setup.py and in requirements.txt

  • ReadMe improvement

  • Updating docs

0.3.1-post2 (2021-05-03)#

  • Bug in NDVI formula, typo when passing to xarray

0.3.1-post1 (2021-04-30)#

  • Correcting a broken link in setup.py

  • Setting a minimum version of sertit

  • Footprint:

    • Fixing a bug in the default function

    • Optimization for S2 and S2-Theia sensors

    • Fixing L7 footprint which was too complex due to nodata stripes

  • CI:

    • Testing footprints and extent

    • Resolving non update of gitlab documentation

    • Changing the resolution of CI processes (to a multiple of the real resolution to speed up the computation)

0.3.1 (2021-04-29)#

  • Multiple optimizations when reading and processing the bands

  • Bug resolutions:

    • S2-Theia masks

    • SAR DSPK bands always recomputed

    • SAR nodata set to 0 as SNAP expects it

    • Bad mask nodata setting when computing invalid pixels

    • Inverted default resolution between L4/5 MSS and TM sensors

  • Removing useless logs when missing DEM but no computed DEM bands

  • Adding copyright headers to every python files

  • Fixing and adding examples

0.3.0 (2021-04-28)#

  • Going Open Source