NWC-SAF is one special SAF because it
develops software. NWC-SAF products are generated locally by
users; that means that it works in the users' side of the
EUMETCast and that there is no bandwidth constraints. For this
reason, NWC-SAF is the nearest SAF to users.
Since GEO-iSHAI is executed locally by
the users, the collocated temperature, specific humidity and ozone
profiles with satellite data can be written in optional binary
files at different steps of the algorithm. The main advantage to
get spatial, temporal and vertically collocated (T, q, ozone)
profiles with satellite data is to allow debugging activities or
generating new parameters or stability indices or generation of 3D
displays as vertical cross sections or 3D visualizations or
generation of validation datasets. These applications are only a
few among other applications that could be used.
At the present time, the generation of
tools for conversion from binary to netCDF format is in progress.
There are a variety of software packages that can read the netCDF
format and can be used by the users to a wider and deeper
exploitation of the GEO-iSHAI binary files.
These conversion tools from GEO-iSHAI
binary files to netCDF format are being generated at the best
effort basis from the IDL prototypes. One possibility it use the
GNU version of IDL called GDL; the IDL prototypes will tested and
adapted to be compatible with GDL if possible. The other
possibility it to develop FORTRAN, C or Python versions; these
could be written using the API netCDF library and the IDL versions
as basis.
The structure of the netCDF format has
been designed to be used directly in application as IDV, McIDAS-V,
IDL, MATLAB, etc. for one easy management of the GEO-iSHAI binary
files for generation of all kinds of visualizations.
As an example, tools freely available as IDV, McIDAS-V could be
used to get all kind of interactive 2D and 3D visualization. As an
example of the possibilities you can see here.
Taking this fact into account, the temperature and humidity profiles interpolated at the 54 RTTOV pressure levels at the main steps of the iSHAI algorithm (after NWP interpolation, FG regressions and physical retrieval) may be written as other optional outputs on binary files. The users can activate the generation easily by editing a keyword in the ASCII configuration file. Then, the binary files will be written in the $SAFNWC/tmp directory. The binary files allows users to debug their local implementation, to get access to the retrieved temperature, humidity and ozone profiles and to compare them with the background NWP profiles
There are two keywords in the
GEO-iSHAI model configuration file to activate writing temperature
and humidity profiles on binary files at different steps of the
algorithm (from the background NWP files supplied by the users to
the final retrieved profiles used to calculate the main outputs).
The users can activate any combination of the two keywords to start writing the optional binary files editing the ASCII GEO-iSHAI model configuration file. Once any of the keywords are activated in the configuration file, the GEO-iSHAI optional binary files will be written in the $SAFNWC/tmp directory in the next execution of GEO-iSHAI. These keywords are:
SAVE_PROFs_in_BINARY_files this keyword activates the writing of background NWP and FG structures. The number of pixels written depends on the value.
1 means the structures are written only at clear air FOR or pixels.
2 means the structures are written in ALL pixels.
SAVE_iSHAI_in_BINARY_file this keywords activates the writing of end profile structures. End profile is the result of physical retrieval or FG non-linear regressions depending on processing options and the convergence of the physical retrieval. The number of pixels written depends on the value.
1 means the structures are written only in clear air FOR or pixels.
2 means the structures are written in ALL pixels.
Using the allowed values of these keywords, users can select to activate the possibilities in Table 1 below.
|
Value = 0 |
Value = 1 |
Value = 2 |
SAVE_PROFs_in_BINARY_files |
Not written |
Written only at clear air FOR or pixels of background NWP and FG profiles |
Written in ALL FOR or pixels of background NWP and FG profiles |
SAVE_iSHAI_in_BINARY_file |
Not written |
Written only at clear air FOR or pixels writing of end profile profiles |
Written in ALL FOR or pixels writing of end profile profiles |
Table 1: combinations for optional binary GEO-iSHAI binary files generation.
Due to the huge size of the GEO-iSHAI optional binary files for large regions (as example full disc region), users should be careful on activation of options when all pixels are written. For this reason it is recommend to use only the option to write at clear air FOR in case of large regions; these files could be used mainly for validation activities.
The content of the optional binary are records that contains C or Fortran-90 structures repeated for every pixel written. Besides the profiles (that means [T[54], q[54], ozone[54], SKT]), the GEO-iSHAI parameters, the synthetic and real satellite BTs, ancillary files, the errors between the real satellite BTs after bias correction and the synthetic RTTOV BTs, etc. are also written. Every record contains now the complete set of parameters of the profile used for the calculations of the GEO-iSHAI outputs. The amount of bytes written in each record depends on the satellite (the number of channels is different in MSG than in Himawari or GOES-R class).
The members of the structures are all float (4 bytes) and the name of the members are described in the tables 2 and 3:
line |
column |
latitude |
longitude |
zenith_angle |
emissivity[nchannels] |
land_sea_type |
topo |
+
bt_satellite[nchannels] |
bias_offset[nchannels] |
bias_scale[nchannels] |
+
bt_rms_threshold |
max_residual |
max_iterations |
emissivity_IR97 |
bt_satellite_IR97 |
bias_offset_IR97 |
bias_scale_IR97 |
Table 2: members in “inputs&ancillary” structure
bt_rttov[nchannels] |
SKT |
psfc |
zsfc |
Tsfc |
qsfc |
p[54] |
T[54] |
q[54] |
O3[54] |
BL |
ML |
HL |
TPW |
LI |
KI |
SHW |
TOZ |
Bt_rms |
residual |
iteration |
qc_flag |
Table 3: members in “iSHAI_profile” structure
The units of q profiles at the 54
RTTOV pressure levels are ppmv. Same for Ozone profile.
The position on region coordinates
(i=element, j=line) are included in every record together with
ancillary parameter (longitude, latitude, etc) in order to
geolocate the FOR.
For BT_SATELLITE
and BT_RTTOV arrays have
dimension nchannels and the order is from lower
to higher wavelength.
The fields on the GEO-iSHAI binary files have been unified for HYB and P mode and just differ in the name of the binary file because the GEO-iSHAI execution mode has been included in the name.
Once any of the keywords are activated in the configuration file, the GEO-iSHAI optional binary files will be written in the $SAFNWC/tmp directory in the next execution of GEO-iSHAI. The GEO-iSHAI optional binary files are always written at $SAFNWC/tmp directory.
Fixing SAVE_iSHAI_in_BINARY_file to 1, the final retrieved profiles (T, q, ozone profile at the 54 RTTOV levels) and other parameters are written only at clear FOR or pixels.
These are the profiles used to calculate TPW, LPW, stability indices and TOZ together with the difference with background NWP profiles. The name of END PROFILE at clear air FOR optional binary file is:
iSHAI_mode_END_____CLEAR_nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin
where:
CLEAR indicates only at clear air FOR
mode processing mode P or HYB
nameregion name of region
Clinc_colc_Ssizl_sizc position of region centre (Clinc_colc) and size of region (Ssizl_sizc)
YYYY-MM-DDThh:mm:00Z is the date (YYYY-MM-DD) and hour (hh:mm:00Z) of SEVIRI image
Example:
iSHAI_HYB_END_____CLEAR_LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin
In this case the binary file is a 1D array [number_clear_FOR] containing values of the structures written on the clear air FOR or pixels where the GEO-iSHAI algorithm has been really executed.
The size of this file is (size_of_record x number_clear_FOR).
Fixing SAVE_iSHAI_in_BINARY_file to 2, the final retrieved profiles (T, q, ozone profile at the 54 RTTOV levels) and other parameters are written for all the pixels. At clear FOR or pixels these profiles are the profiles used to calculate GEO-iSHAI parameters and for cloudy or not processed pixels dummy structures filled all members with missing value (-9999.0) are written.
The name of END PROFILE at all pixels optional binary file is:
iSHAI_mode_END_____nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin
where:
Mode processing mode P or HYB
Nameregion name of region
Clinc_colc_Ssizl_sizc position of region centre (linc, colc are the line and column coordinates of the centre of the region (0-based)) and size of region (Ssizl_sizc are the size of the region (number of lines and number of columns)).
YYYY-MM-DDThh:mm:00Z is the date (YYYY-MM-DD) and hour (hh:mm:00Z) of SEVIRI image
Example:
iSHAI_HYB_END_____LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin
In this example LargeIberica is a region of 665 lines x 511 columns centred in (558, 1758) by this reason the name of binary file contains C1748_0558_S0665_0511
In this case the binary file is a 2D array [ncols, nlines] containing values of the structures written on all pixels. These binary files are adequate for small to medium regions when processing in 1x1 FOR and in applications as vertical cross sections, 3D visualizations, etc.
Note: users must take into account the huge size of the binary files that could be generated and the very expensive CPU time of the pixel by pixel processing for large regions.
The size of this file is (size_of_record x ncols x nlines)
Fixing SAVE_PROFs_in_BINARY_files to 1, the background NWP and after FG non-linear regressions profiles (T, q, ozone profile at the 54 RTTOV levels) and other parameters are written only at clear FOR or pixels. These are the profiles used to calculate TPW, LPW, stability indices and TOZ together of the difference with the end GEO-iSHAI profiles. The name of background nwp and FG at clear air FOR optional binary files are respectively:
iSHAI_mode_nwp_____CLEAR_nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin
iSHAI_mode_FG______CLEAR_nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin
where:
CLEAR indicates only at clear air FOR
Mode processing mode P or HYB
Nameregion name of region
Clinc_colc_Ssizl_sizc position of region centre (Clinc_colc) and size of region (Ssizl_sizc)
YYYY-MM-DDThh:mm:00Z is the date (YYYY-MM-DD) and hour (hh:mm:00Z) of SEVIRI image
Examples:
iSHAI_HYB_nwp_____CLEAR_LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin
iSHAI_HYB_FG______CLEAR_LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin
In this case the binary files are a 1D array [number_clear_for] containing values of the structures written on the clear air FOR or pixels where the GEO-iSHAI algorithm has been really executed.
The size of this file is (size_of_record x number_clear_FOR).
Fixing SAVE_PROFs_in_BINARY_files to 2, the background NWP and after FG non-linear regressions profiles (T, q, ozone profile at the 54 RTTOV levels) and other parameters are written for all the pixels. These are the profiles used to calculate GEO-iSHAI parameters at clear FOR or pixels and dummy structures filled all members with missing value (-9999.0) for cloudy or not processed pixels.
The name of background nwp and FG at all pixels optional binary files are respectively:
iSHAI_mode_nwp_____nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin
iSHAI_mode_FG______nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin
where:
Mode processing mode P or HYB
Nameregion name of region
Clinc_colc_Ssizl_sizc position of region centre (Clinc_colc) and size of region (Ssizl_sizc)
YYYY-MM-DDThh:mm:00Z is the date (YYYY-MM-DD) and hour (hh:mm:00Z) of SEVIRI image
Examples:
iSHAI_HYB_nwp_____LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin
iSHAI_HYB_FG______LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin
In this case the binary file are a 2D
array [ncols, nlines] containing values of the structures written
on all pixels. It is more useful when processing in 1x1
FOR.
The size of this file is (size_of_record x ncols x nlines)
Note: users must take into account the huge size of the binary files that could be generated and the very expensive CPU time of the pixel by pixel processing for large regions.
The binary files are not
easy to manage and since 2010 some IDL procedures have been
developed to convert the iSHAI binary files to netCDF format.
In order to facilitate
the use of the binary files, tools to convert the GEO-iSHAI binary
files to netCDF format has being developed adapting the IDL
prototypes available from former PGE13 SPhR ones. These conversion
tools from GEO-iSHAI binary files to netCDF format are being
generated at the best effort basis from the IDL prototypes. One
possibility it use the GNU version of IDL called GDL; the IDL
prototypes will tested and adapted to be compatible with GDL if
possible. The other possibility it to develop FORTRAN, C or Python
versions; these could be written using the API netCDF library and
the IDL versions as basis.
There are a great variety of software packages that can also read the netCDF format and can be used by the users to a wider and deeper exploitation of the GEO-iSHAI binary files. Among all software packages IDL, MATLAB, GDAL, McIDAS-V, IDV, Python netCDF reader, etc. could be used for easy management of the GEO-iSHAI binary files and/or netCDF for validation and/or visualization purposes.
In order to show examples of the important applications for monitoring of change using these optional binary files, tools freely available as IDV, McIDAS-V could be used as example to get all kind of interactive 2D and 3D visualization. See the vertical cross sections on the case study included in the visualization examples.In the link several examples of the use of vertical cross sections for nowcasting for a case study is presented. For more examples of the possibilities you can see also the Martinez presentations and links in the iSHAI references page.