For really huge COGs (let's say 1 million pixel x 1 million pixel), the size of the TileByteCount and TileOffsets values can be quite big (several tens of MB). If one knows that the tiles are ordered in a top-to-bottom, left-to-right order in the file, then a smart reader can possibly only use TileOffsets, and that's what GDAL does, to save a GET request. To do so, one must write the header ghost area per https://gdal.org/drivers/raster/cog.html#header-ghost-area and add a 4-byte header before each tile data and a 4-byte trailer after it (https://gdal.org/drivers/raster/cog.html#tile-data-leader-and-trailer), whose purpose is to check that a writer hasn't modified tile content since the COG generation, which would make the optimization no longer valid.

The read me says both "internally tiled geotiff (optionally with overviews" and "it should contain overviews". The statements are at odds with each other. Which is the more correct meaning?

Hi,

Thanks a lot for this amazing tool ! I have just converted a 15Go ECW to GeoTiff with GDAL. To be able to use the YCBCR jpeg optimization, I have been "forced" to remove the 4th transparency band of the ECW. I used the -mask 4 to keep the transparency and avoid to have a big black rectangle.

gdalinfo against the ecw gives: https://paste.sh/DAITkjGJ#P0ui5RBQaEDpDP0vQnHSWyny

My use of gdal_translate & gdaladdo:

# convert ecw to tif
gdal_translate -a_srs EPSG:2154 -r nearest -co COMPRESS=JPEG -co JPEG_QUALITY=60 --config INTERLEAVE PIXEL -co PHOTOMETRIC=YCBCR -b 1 -b 2 -b 3 -mask 4 -co TILED=YES -co BIGTIFF=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 --config GDAL_CACHEMAX 500 --config GDAL_MAX_DATASET_POOL_SIZE 1014 -co NUM_THREADS=2 -of GTiff orthophoto.ecw orthophoto.tif

# pyramids
gdaladdo -r average -ro --config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 60--config INTERLEAVE_OVERVIEW PIXEL --config PHOTOMETRIC_OVERVIEW YCBCR --config TILED_OVERVIEW YES --config GDAL_TIFF_OVR_BLOCKSIZE 512 --config BIGTIFF_OVERVIEW YES --config GDAL_CACHEMAX 500 --config GDAL_MAX_DATASET_POOL_SIZE 1014 -co NUM_THREADS=2 orthophoto.tif 2 4 8 16 32 64 128 256

The output Geotiff is exactly what I expected, but with this -b 1 -b 2 -b 3 -mask 4 technique, I have four files: orthophoto.tif, orthophoto.tif.ovr, orthophoto.tif.msk, orthophoto.tif.msk.ovr

Then I wanted to use cogger with

./cogger -output orthophoto.cog.tif orthophoto.tif orthophoto.tif.ovr

I get the expected orthophoto.cog.tif file, but the mask has not been "integrated" into the cog, so I get the black rectangle around the data, for example when viewing this COG file inside QGIS.

Do I need to create a new COG file from the orthophoto.tif.msk and orthophoto.tif.msk.ovr files, or is there a better way ?

• The gdal ghost area optimisations can be disabled, and will not be emitted for PlanarConfiguration=Separate, as it is not supported by gdal. Fixes #8
• Each IFD may now reference only a single mask
• The IFD structure is now exported and can be used to hand-craft an expected COG from dynamic data
• Encoding (little vs. big endian) is now configurable
• For PlanarConfiguration=Separate files, it is possible to fine tune the layout/interlacing of each plane's tiles and mask tiles, supporting tile orders like r1,g1,b1,r2,g2,b2 (the default), or r1,r2...g1,g2...b1,b2 or even r1,g1,r2,g2,...rn,gn,b1,b2

planarconfiguration=separate is not an officially recognized configuration for COGs, which is an oversight as the same principles as for planarconfiguration=contig also apply in this case. cogger should probably not emit a ghost area for those type of files so as not to confuse a parser using that information. c.f. opengeospatial/CloudOptimizedGeoTIFF#3 and OSGEO/gdal#4931