CryoClim - npolar/RemoteSensing GitHub Wiki

Project Info

Webside NP: http://www.cryoclim.net/cryoclim/index.php/Svalbard_glacier_products

File Location

Files processed prior to Sept 2012: Z:\Svalbard\CryoClimValidation\NestBeforeSept2012

Original SAR files: Z:\Svalbard\ERS1_2_ASAR

Working Files after Sept 2012: C:\Users\max\Documents\Svalbard\CryoClimValidation

Backup of Working Files after Sept 2012: Z:\Svalbard\CryoClimValidation<br />

Inventory in C:\Users\max\Documents\Svalbard\SatelliteData\AvailableSARdata.xlsx

Scripts located in C:\Users\max\Documents\Svalbard\CryoClimValidation

Datasets on \hjalmar\PublicDatasets\Datasets\cryoclim which is accessible through http://public.data.npolar.no/cryoclim/

Chosen Glaciers and Images###

'''Validation Glaciers:'''

Kongsvegen --> SARSIM images ready
Hansbreen --> SARSIM images ready
Hayesbreen/Ulvebreen --> SARSIM images ready
NortheastSpitsbergen --> SARSIM images ready 1992-2002
Etonbreen --> SARSIM images ready, image selection not completely satisfying
Edgeøya --> SARSIM images ready
Northwest (Monacobreen) --> SARSIM images ready
Fridtjofbreen --> analyse available images

'''Kongsvegen''' no images in EOLI after Dec 2011

File Path C:\Users\max\Documents\Svalbard\CryoClimValidation\Kongsfjorden
Backup Z:\Svalbard\CryoClimValidation\Kongsfjorden
1992	ERS-1	30.04.1992	481	1989	4133
1993	ERS-1	04.02.1993	481	1989	8141
1994	ERS-1	16.12.1993	481	1989	12650
1995	ERS-1	08.04.1995	481	1989	19507
1996	ERS-1	17.02.1996	481	1989	24016
1997	ERS-2	09.03.1997	481	1989	9854
1998	ERS-2	29.03.1998	481	1989	15365
1999	ERS-2	01.05.1999	166	1989	21062
2000	ERS-2	27.02.2000	481	1989	25385
2001	ERS-2	07.01.2001	481	1989	29894
2002	ERS-2	03.03.2002	481	1989	35906
2003	ERS-2	23.03.2003	481	1989	41417
2004	ASARAPP	07.03.2004	481	1989	
2005	ASARIMG	08.03.2005			
2006	ASARAPP	16.04.2006	481	1989	21577
2007	ERS-2	25.02.2007	481	1989	61985
2008	ASARAPP	10.02.2008	481	1989	31069
2009	ASARIMG	01.03.2009		
2010	ERS-2	21.03.2010	481	1989	77990
2011		26.12.2010			
2012		none

'''Hansbreen''' 1992 and 2007 missing, would be available as ascending, no images after 2010

File Path C:\Users\max\Documents\Svalbard\CryoClimValidation\SouthSpitsbergen
Backup Z:\Svalbard\CryoClimValidation\SouthSpitsbergen

1993	ERS-1	08.02.1993	2037	2025	8190
1994		19.04.1994			
1995	ERS-1	12.02.1995	1917	2025	18718
1996	ERS-1	05.03.1996	2223	2025	24259
1997	ERS-2	17.04.1997	2037	2025	10412
1998	ERS-2	04.02.1998	2223	2025	14606
1999	ERS-2	12.03.1999			
2000	ERS-2	25.02.2000			
2001	ERS-2	09.02.2001			
2002	ERS-2	01.03.2002	452	2025	35877
2003	ASARAPP	05.03.2003			
2004	ASARAPP	24.03.2004	2223	2025	10789
2005	ASARAPP	09.03.2005			
2006	ASARAPP	04.04.2006	2309	2025	5205
2007		none			
2008		07.01.2008			
2009		04.11.2008	at edge of image		
2010		none			
2011		none			
2012		none

'''Hayesbreen / Ulvebreen'''

File Path C:\Users\max\Documents\Svalbard\CryoClimValidation\CentralSpitsbergen\AppOrb_Calib_Spk_SarsimTC_LinDB
Backup Z:\Svalbard\CryoClimValidation\CentralSpitsbergen
chosen images			visual check	
				
1992		20.10.1991	ok	
1993		08.02.1993	ok	
1994		28.11.1993	ok	
1995		12.02.1995	ok	
1996		05.03.1996	ok	
1997		17.04.1997	ok	
1998		04.02.1998	ok	
1999		19.12.1998	ok	
2000		14.10.1999	ok	
2001		26.04.2001	ok	
2002		07.03.2002	ok	
2003		05.03.2003	ok	
2004		05.02.2004	ok	
2005		31.03.2005	ok	
2006		27.10.2005	ok	
2007		none		
2008		06.12.2007	ok 	not Hayes!
2009		none		
2010		none		
2011		none		
2012		none

'''Etonbreen''' 21 Sept 2012: Images not satisfying, Etonbreen too close at edge for many years

File Path C:\Users\max\Documents\Svalbard\CryoClimValidation\Nordaustlandet
Backup Z:\Svalbard\ERS1_2_ASAR\Nordaustlandet

						in archive	
1992		24.03.1992	452	1971	in corner	ok	
1993		15.03.1993				ok	
1994		28.11.1993	223	1971		ok	
1995		12.02.1995				ok	
1996		05.03.1996				ok	
1997		17.04.1997				ok	
1998		06.01.1998				ok	
1999		03.12.1998				ok	12.3.1999 front missing
2000		11.10.1999				ok	
2001		11.10.2000				ok	
2002		27.12.2001			in corner	ok	
2003		05.03.2003				ok	
2004		24.03.2004	223	1971		ok	
2005		09.03.2005	223	1971		ok	
2006		07.04.2006			too much in corner	ok	
2007		available none ordered					
2008		19.12.2007				ok	
2009		27.03.2009			too much in corner	ok	
2010		20.10.2009	309		in corner	ok	
2011		24.03.2011			at edge	ok	
2012

'''Northeastern Spitsbergen (Glacier to be chosen)'''

File Path C:\Users\max\Documents\Svalbard\CryoClimValidation\NortheastSpitsbergen
Backup Z:\Svalbard\CryoClimValidation\NortheastSpitsbergen
				
				visual check
1992	26.02.1992			ok
1993	08.02.1993			ok
1994	13.04.1994			ok
1995	26.02.1995			ok
1996	22.02.1996			ok
1997	17.04.1997			ok
1998	22.01.1998			ok
1999	03.12.1998			ok
2000	18.03.2000			ok
2001	26.04.2001			ok
2002	27.12.2001			ok
2003	14.3.2003/5.3.		mosaic	ok
2004	27.2.2004 /27.3.		mosaic	ok
2005	none			
2006	none			
2007	none			
2008	several			
2009	none			
2010	limited area coverage			
2011	limited area coverage			
2012	none

'''Edgeøya'''

File Path C:\Users\max\Documents\Svalbard\CryoClimValidation\EdgeoyaBarentsoya
Backup Z:\Svalbard\CryoClimValidation\EdgeoyaBarentsoya

						in archive
						
1992	01.03.1992					ok
1993	23.04.1993					ok
1994	02.01.1994					ok
1995	04.02.1995		mosaic			ok
1996	12.03.1996					ok
1997	none					
1998	26.01.1998					ok
1999	30.01.1999/6.05.	east missing -- mosaic 	ok
2000	15.01.2000		east missing,not avail.	ok
2001	26.03.2001		some east missing	ok
2002	04.02.2002					ok
2003	31.03.2003					ok
2004	15.03.2004		mosaic			ok
2005	28.02.2005					ok
2006	none					        ok
2007	14.03.2007		east  not avail.	ok
2008	10.12.2007					ok
2009	none					
2010	none					
2011	13.03.2011					ok
2012	none

'''Northwest / Monacobreen''' file path C:\Users\max\Documents\Svalbard\CryoClimValidation\NorthwestSpitsbergen backup Z:\Svalbard\CryoClimValidation\NorthwestSpitsbergen

	
1992	11.03.1992
1993	26.02.1993
1994	12.04.1994
1995	22.02.1995
1996	24.02.1996
1997	15.03.1997
1998	28.02.1998
1999	05.12.1998
2000	04.03.2000
2001	24.03.2001
2002	02.02.2002
2003	29.03.2003
2004	13.03.2004
2005	none
2006	25.04.2006
2007	03.03.2007
2008	08.12.2007
2009	11.12.2008
2010	16.01.2010
2011	13.12.2010
2012	none

Processing line for the SAR images

###= Import to DIM ###=

use this bat file to import:

:Batch file reading all files in directory and run NEST script :check http://www.computerhope.com/forhlp.htm :for gpt check http://www.array.ca/nest-web/help/tutorials/commandLineProcessing.html :Ending E1 Ers1, E2 ERS2, old format open VDF_DAT.001, N1 -- ASAR APP for /r Z:\Svalbard\ERS1_2_ASAR\Kongsfjorden %%X in (*.E1) do ( echo processing %%~nX gpt C:\Users\max\Documents\Svalbard\CryoClimValidation\ImportToDIM.xml -Pfile="%%X" -Tfile="C:\Users\max\Documents\Svalbard\CryoClimValidation\Kongsfjorden\NestImported%%~nX.dim" )

For the VDF files get the filename from the path name:

for /r Z:\Svalbard\ERS1_2_ASAR\Kongsfjorden %%X in (VDF_*) do ( echo processing %%X for /f "delims=\ tokens=1,2,3, 4, 5, 6" %%i in ("%%X") do ( gpt C:\Users\max\Documents\Svalbard\CryoClimValidation\ImportToDIM.xml -Pfile="%%X" -Tfile="C:\Users\max\Documents\Svalbard\CryoClimValidation\Kongsfjorden\NestImported%%m.dim" ))

the XML file is:

1.0 Read $file Write BEAM-DIMAP $target

Important: Earlier versions have instead!!

#####= Apply Orbit File ###= not possible for all files:
ERS1 orbits are available until 1996
ERS2 orbits are available until 2003
ENVISAT orbits are available until 2008

If problems with accessing Orbit files automatically:

Dear Max, The address is correct as you can verify from the the Settings window you can access from (from EDIT--> Settings).

However I have the same problem here in Esrin and also myself (from Esrin) was able to download successfully the orbit and the DEMs few times ago. I have tried from home and it works, then I am convinced that it is a problem coming from the internet firewall setting which indeed is centralised and managed by the system administrator.

My solution has been to download all the orbit manually and put locally in the directory created by NEST by default :

C:\AuxData\Orbits\Delft Precise Orbits\ODR.ENVISAT1\eigen-cg03c

Best wishes, Andrea

Bat File for /r C:\Users\max\Documents\Svalbard\CryoClimValidation\Kongsfjorden\NestImported %%X in (*.dim) do (gpt C:\Users\max\Documents\Svalbard\CryoClimValidation\AppOrb.xml -Pfile="%%X"
-Tfile="C:\Users\max\Documents\Svalbard\CryoClimValidation\Kongsfjorden\AppOrb%%~nX_AppOrb.dim")

XML File 1.0 Read $file Write BEAM-DIMAP $target Apply-Orbit-File DELFT Precise (ENVISAT, ERS1&2)

###= Calibration, Speckle, Terrain Correction, Linear to dB ###=

The version in this link is faster than SARSIM_TC but fails at present for some images. SARSIM and TC gives almost identical results, so one option to process failed images separately. Following the other option: For consistency processing all with SARSIM.

Issue: Some images are so offset apparently (orbit parameters) that the correlation does not catch in TC

###= Calibration, Speckle, SARSIMTerrain Correction, Linear to dB ###=

Polarisations need to be processed separately. Must be repeated for Sigma0_HH for files where Sigma0_VV does not exist, change in XML file.
ca 18 minutes per scene
If error message that 0 GCPs survived, bigger coarse reg window or more GCP in XML file.

Batch file for /r C:\Users\max\Documents\Svalbard\CryoClimValidation\Kongsfjorden\AppOrb %%X in (*.dim) do (gpt C:\Users\max\Documents\Svalbard\CryoClimValidation\Calib_Spk_SarsimTC_LinDB.xml -Pfile="%%X"
-Tfile="C:\Users\max\Documents\Svalbard\CryoClimValidation\Kongsfjorden\AppOrb_Calib_Spk_TC_LinDB%%~nX_Calib_Spk_TC_LinDB.dim")

And the XML-file, '''the DEM file must be changed for other regions!'''

1.0 Read $file Calibration Latest Auxiliary File false false false false Speckle-Filter Median 3 3 2 5000.0 true 1.0 SAR-Simulation Sigma0_VV External DEM BILINEAR_INTERPOLATION C:\Users\max\Documents\Svalbard\GIS\DTM\dem20subsetNW.tif 9999.0 false GCP-Selection 600 512 512 2 2 2 0.5 false 128 128 3 0.6 false SARSim-Terrain-Correction 1.0 1 BILINEAR_INTERPOLATION 20 1.796630568239043E-4 PROJCS["UTM Zone 33 / World Geodetic System 1984", GEOGCS["World Geodetic System 1984", DATUM["World Geodetic System 1984", SPHEROID["WGS 84", 6378137.0, 298.257223563, AUTHORITY["EPSG","7030"]], AUTHORITY["EPSG","6326"]], PRIMEM["Greenwich", 0.0, AUTHORITY["EPSG","8901"]], UNIT["degree", 0.017453292519943295], AXIS["Geodetic longitude", EAST], AXIS["Geodetic latitude", NORTH]], PROJECTION["Transverse_Mercator"], PARAMETER["central_meridian", 15.0], PARAMETER["latitude_of_origin", 0.0], PARAMETER["scale_factor", 0.9996], PARAMETER["false_easting", 500000.0], PARAMETER["false_northing", 0.0], UNIT["m", 1.0], AXIS["Easting", EAST], AXIS["Northing", NORTH]] false false false true false false false false Use projected local incidence angle from DEM Use projected local incidence angle from DEM Latest Auxiliary File false false LinearTodB Write BEAM-DIMAP $target

Creating Masks for Glacier Processing

Add GLIMS layer in ArcGis; saved separate now as C:\Users\max\Documents\Svalbard\CryoClimValidation\GlacierMasks_2000s.lyr
Select glacier and choose right-click>export
Use Buffer(Analysis) Tool to decrease polygon. Necessary to avoid effects at edges.

Analysis of GBA Validation

K-means (and GST) processing is stored in C:\Users\max\Documents\Svalbard\CryoClimValidation\FinalProducts -- Files can be nicely imported to Igor Pro

Graphs stored in C:\Users\max\Documents\Svalbard\CryoClimValidation\CryoClimValidation.pxp Igor Pro file

Trying to normalize kmeans value as a sort of AAR: "The [http://accumulation%20area%20ratio http://en.wikipedia.org/wiki/Tidewater_glacier_cycle] of a glacier, AAR, is the percentage of a glacier that is a snow-covered accumulation zone at the end of the summer melt season."

"AAR" = kmeans/Total

Totals (first column of image, contains outside area, second total of glacier): Aavatsmarkbreen 265024 83450 Chydeniusbreen 1848825 253795 Comfortlessbreen 264966 64194 Elisebreen 40898 18025 Etonbreen 1692702 824119 Hansbreen 216733 70920 Hayesbreen 490200 85413,3 Irenebreen 11349 4026,2 Kongsvegen 680680 189437 Kuhrbreen 374010 152409 Raudfjordbreen 304158 50425 Ulvebreen 220978 87175 Uversbreen 451260 78259 Veteranbreen 753452 232953 Waldemarbreen 7125 2520

GST workflow

buffer glacier mask at edges (manually before script or within)
rasterize glacier mask, size bounding box
crop SAR image at bounding box
remove values outside mask
Otsu classification -- try Otsu3 in http://code.google.com/p/python-microscopy/source/browse/cpmath/otsu.py?spec=svn723c7e28f1385990003d5994605f5c096bdd2568&r=723c7e28f1385990003d5994605f5c096bdd2568
create classified raster based on thresholds (one for each surface type?)
make polygon out of raster (how?)

Matlab flow path

Cryoclim flow in Matlab

Date: 2012-05-07

###=Flow consists of two bulks:###=

read/process/save. This reads all data in folder and subfolders. Therefore keep all the *.dim files for a particular glacier in one folder. It will then create one output file for each glacier and parameter (i.e. glacier surface type and glacier balance area (raster)) containing all data for that glacier.
export to netCDF and make metadata. This creates (if all data for one glacier is used) a netDCF file for that glacier containing the chosen parameter (i.e. glacier surface type or glacier balance area (raster)) as well as one metadata file (DIF).

###=Bulk 1:###=

Executed using function BeamDimap.m

Following input needed (example parameters):

crop polygon name (*.kml file containing polygon):

polyName = 'Hayes.kml';

and

Choose directory containing *.dim files in popup window opened when executing function (not it will also search all subfolders!).

BeamDimap.m will read all *.dim files in given directory and process and save data using the following functions:

BeamDimapReader_cryoclim.m read *.dim files within directory. Files must be in UTM33 standard BeamDimap format from NEST.

BeamDimapPolyCrop.m masks out all data outside of polygon.

BeamCropArea.m minimizes the frame area to polygon.

BeamDimapKMeans.m makes the k-means classification.

GlacierSurfaceType.m classifies surface type based on backscatter.

The surface type, k-means and backscatter is appended to the respective output files:

C:\Users\ola\Documents\MATLAB\Cryoclim<nowiki>SAR_Glaciername_surf.mat

C:\Users\ola\Documents\MATLAB\Cryoclim<nowiki>SAR_Glaciername_kmeans.mat

C:\Users\ola\Documents\MATLAB\Cryoclim<nowiki>SAR_Glaciername_data.mat

using Beam2mat.m.

Note: All the scenes get stored in a 3D matrix where the 3rd dimension is time (first and second is easting and northing). The time for each layer in the 3D matrix is stored in the time variable.

###=Bulk 2:###=

The output files from Bulk 1 are read into Beam2netCDF_v2.m. Set which glacier to process. For example:

GlacierName = 'Kongsvegen';

Also set what data type you want to process i.e. GST or GBA:

Type2Process = 'GST'; or Type2Process = 'GBA'

Name of exported *.nc files for thredds will be (example):

'SAR_Kongsvegen_kmeans.nc' for GBA

'SAR_Kongsvegen_surf.nc' for GST

NOTE!! There is a section in the script that removes certain scenes. That is done by manually input what scene you want removed. That will likely need to be changed if you change the number of files in the folder. [MAX: this was commented ot in my recent version] Metadata in DIF format are made within the function using Metadata2Dif.m. Naming will be something like:

'Cryoclim_GBA_Kongsvegen_2006-04-16-2010-03-21.xml'

'Cryoclim_GST_Kongsvegen_2006-04-16-2010-03-21.xml'

You can validate the *.nc files using for example 'Panoply'. Note you really need to zoom in to see the data. Zoom by first mark data and click 'Create plot'. That creates the plot... Go to 'Map' and set to 'Equirectangular (Regional)' Enter coordinates and wait...

[Max: The original script did creat metadat before sorting the dates, so wrong start date in metadata -- fixed. I added code making a textfile with numbers of pixels per class. THe shapefile masks seems to give slightly different area size for glacier area in different years]

Export netCDF to thredds

Files on thredds are found here:

http://thredds.data.npolar.no/thredds/catalog/testAll/catalog.html

See here for information about thredds on Bjarne:

http://wiki.npolar.no/wiki/index.php/Software_on_bjarne#Thredds_-_Cryoclim

metadata (DIF) for the netCDF files

Are made using the matlab work flow.

Not system setup to export to met.no's metamod. Now sent by mail to Øystein G and met.no

Move *.nc files to Bjarne

from Ola Brandt: I have chosen to move the *.nc files to Bjarne using 'WinSCP'. log into bjarne.npolar.no use appropriate user and password.

You find the folder by going up one level to "opt" and then to

"apache-tomcat-6.0.24/content/thredds/public/cryoclim"

See data here: