Cleaning methods when loading optical bands

Let’s take a peek on the cleaning methods of optical bands and their potential respective time-consumption.

Note that these keywords are working with both load and stack functions.

Try with Landsat-8

Let’s open a Landsat-8 OLCI collection 2 tile. Landsat COL-2 products manage their nodata and defective pixels through two flag files:

  • QA_PIXELS

  • QA_RADSAT

See more about these files here

# Imports
import os
from eoreader.reader import Reader
from eoreader.bands import GREEN
from eoreader.keywords import CLEAN_OPTICAL
from eoreader.products import CleanMethod
# Open the product
folder = os.path.join("/home", "data", "DS3", "CI", "eoreader", "optical")
path = os.path.join(folder, "LC08_L1TP_200030_20201220_20210310_02_T1.tar")
eoreader = Reader()
prod = eoreader.open(path)

Time the RAW method

The RAW method is simple: just open the given tile with no pixel processing.

%%timeit
prod.load(
    GREEN, 
    **{CLEAN_OPTICAL: CleanMethod.RAW}
)
prod.clean_tmp()
1.8 s ± 41.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Time the NODATA method

Only the detector nodata is processed by the NODATA method.
The bands will be set to nodata outside of the detector footprint (instead of keeping the raw nodata value)

%%timeit
prod.load(
    GREEN, 
    **{CLEAN_OPTICAL: CleanMethod.NODATA}
)
prod.clean_tmp()
2.79 s ± 133 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Time the CLEAN method

Every defective pixel given by the provider by the CLEAN method. These pixels will be set to nodata.

%%timeit
prod.load(
    GREEN, 
    **{CLEAN_OPTICAL: CleanMethod.CLEAN}
)
prod.clean_tmp()
5.02 s ± 85.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Try another product: Sentinel-2

Let’s open a Sentinel-2 (processing baseline < 04.00, ~acquired before end of 2021, with flag files provided as vectors).

The invalid pixel are retrived from the files:

  • DETFOO: Detector footprint (nodata outside the detectors)

  • NODATA: Pixel nodata (inside the detectors) (QT_NODATA_PIXELS)

  • DEFECT: Defective pixels

  • SATURA: Saturated Pixels

  • TECQUA: Technical quality mask (MSI_LOST, MSI_DEG)

# Open the product
path = os.path.join(folder, "S2B_MSIL2A_20200114T065229_N0213_R020_T40REQ_20200114T094749.SAFE")
eoreader = Reader()
prod = eoreader.open(path)

Time the RAW method

The RAW method is simple: just open the given tile with no pixel processing.

%%timeit
prod.load(
    GREEN,
    **{CLEAN_OPTICAL: CleanMethod.RAW}
)
prod.clean_tmp()
1.77 s ± 53.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Time the NODATA method

Only the detector nodata is processed by the NODATA method.
The bands will be set to nodata outside of the detector footprint (instead of keeping the raw nodata value)

%%timeit
prod.load(
    GREEN,
    **{CLEAN_OPTICAL: CleanMethod.NODATA}
)
prod.clean_tmp()
2.2 s ± 29.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Time the CLEAN method

Every defective pixel given by the provider by the CLEAN method. These pixels will be set to nodata.

%%timeit
prod.load(
    GREEN,
    **{CLEAN_OPTICAL: CleanMethod.CLEAN}
)
prod.clean_tmp()
2.71 s ± 65.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Try with the latest Sentinel-2 baseline

Let’s open a Sentinel-2 (processing baseline >= 04.00, ~acquired after end of 2021, with flag files provided as rasters).

The invalid pixel are retrived from the file:

  • QUALIT: Regrouping TECQUA, DEFECT, NODATA, SATURA

The nodata pixels (outside detector footprints) are now retrieved from null pixels, as a radiometric offset has been added.

See here for more information about the processing baseline update.

# Open the product
path = os.path.join(folder, "S2B_MSIL2A_20210517T103619_N7990_R008_T30QVE_20211004T113819.SAFE")
eoreader = Reader()
prod = eoreader.open(path)

Time the RAW method

The RAW method is simple: just open the given tile with no pixel processing.

%%timeit
prod.load(
    GREEN,
    **{CLEAN_OPTICAL: CleanMethod.RAW}
)
prod.clean_tmp()
1.85 s ± 34.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Time the NODATA method

Only the detector nodata is processed by the NODATA method.
The bands will be set to nodata outside of the detector footprint (instead of keeping the raw nodata value)

%%timeit
prod.load(
    GREEN,
    **{CLEAN_OPTICAL: CleanMethod.NODATA}
)
prod.clean_tmp()
1.99 s ± 42.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

Time the CLEAN method

Every defective pixel given by the provider by the CLEAN method. These pixels will be set to nodata.

%%timeit
prod.load(
    GREEN,
    **{CLEAN_OPTICAL: CleanMethod.CLEAN}
)
prod.clean_tmp()
23.3 s ± 1.34 s per loop (mean ± std. dev. of 7 runs, 1 loop each)