Optional iSHAI binary files.

NWC-SAF is one special SAF because it develops software. NWC-SAF products are generated locally by users; that means that it works in the users' side of the EUMETCast and that there is no bandwidth constraints. For this reason, NWC-SAF is the nearest SAF to users.

Since GEO-iSHAI is executed locally by the users, the collocated temperature, specific humidity and ozone profiles with satellite data can be written in optional binary files at different steps of the algorithm. The main advantage to get spatial, temporal and vertically collocated (T, q, ozone) profiles with satellite data is to allow debugging activities or generating new parameters or stability indices or generation of 3D displays as vertical cross sections or 3D visualizations or generation of validation datasets. These applications are only a few among other applications that could be used.

At the present time, the generation of tools for conversion from binary to netCDF format is in progress. There are a variety of software packages that can read the netCDF format and can be used by the users to a wider and deeper exploitation of the GEO-iSHAI binary files.

These conversion tools from GEO-iSHAI binary files to netCDF format are being generated at the best effort basis from the IDL prototypes. One possibility it use the GNU version of IDL called GDL; the IDL prototypes will tested and adapted to be compatible with GDL if possible. The other possibility it to develop FORTRAN, C or Python versions; these could be written using the API netCDF library and the IDL versions as basis.

The structure of the netCDF format has been designed to be used directly in application as IDV, McIDAS-V, IDL, MATLAB, etc. for one easy management of the GEO-iSHAI binary files for generation of all kinds of visualizations.
As an example, tools freely available as IDV, McIDAS-V could be used to get all kind of interactive 2D and 3D visualization. As an example of the possibilities you can see here.

Taking this fact into account, the temperature and humidity profiles interpolated at the 54 RTTOV pressure levels at the main steps of the iSHAI algorithm (after NWP interpolation, FG regressions and physical retrieval) may be written as other optional outputs on binary files. The users can activate the generation easily by editing a keyword in the ASCII configuration file. Then, the binary files will be written in the $SAFNWC/tmp directory. The binary files allows users to debug their local implementation, to get access to the retrieved temperature, humidity and ozone profiles and to compare them with the background NWP profiles

Activation and generation of optional iSHAI binary files.

There are two keywords in the GEO-iSHAI model configuration file to activate writing temperature and humidity profiles on binary files at different steps of the algorithm (from the background NWP files supplied by the users to the final retrieved profiles used to calculate the main outputs).

The users can activate any combination of the two keywords to start writing the optional binary files editing the ASCII GEO-iSHAI model configuration file. Once any of the keywords are activated in the configuration file, the GEO-iSHAI optional binary files will be written in the $SAFNWC/tmp directory in the next execution of GEO-iSHAI. These keywords are:

SAVE_PROFs_in_BINARY_files this keyword activates the writing of background NWP and FG structures. The number of pixels written depends on the value.

1 means the structures are written only at clear air FOR or pixels.

2 means the structures are written in ALL pixels.

SAVE_iSHAI_in_BINARY_file this keywords activates the writing of end profile structures. End profile is the result of physical retrieval or FG non-linear regressions depending on processing options and the convergence of the physical retrieval. The number of pixels written depends on the value.

1 means the structures are written only in clear air FOR or pixels.

2 means the structures are written in ALL pixels.

Using the allowed values of these keywords, users can select to activate the possibilities in Table 1 below.

	Value = 0	Value = 1	Value = 2
SAVE_PROFs_in_BINARY_files	Not written	Written only at clear air FOR or pixels of background NWP and FG profiles	Written in ALL FOR or pixels of background NWP and FG profiles
SAVE_iSHAI_in_BINARY_file	Not written	Written only at clear air FOR or pixels writing of end profile profiles	Written in ALL FOR or pixels writing of end profile profiles

Table 1: combinations for optional binary GEO-iSHAI binary files generation.

Due to the huge size of the GEO-iSHAI optional binary files for large regions (as example full disc region), users should be careful on activation of options when all pixels are written. For this reason it is recommend to use only the option to write at clear air FOR in case of large regions; these files could be used mainly for validation activities.

The content of the optional binary are records that contains C or Fortran-90 structures repeated for every pixel written. Besides the profiles (that means [T[54], q[54], ozone[54], SKT]), the GEO-iSHAI parameters, the synthetic and real satellite BTs, ancillary files, the errors between the real satellite BTs after bias correction and the synthetic RTTOV BTs, etc. are also written. Every record contains now the complete set of parameters of the profile used for the calculations of the GEO-iSHAI outputs. The amount of bytes written in each record depends on the satellite (the number of channels is different in MSG than in Himawari or GOES-R class).

For all the files and for every pixel (or FOR) the structures inputs&ancillary and iSHAI_profile declared in $SAFNWC/src/iSHAI/iSHAI_*/iSHAI.c are written.

The members of the structures are all float (4 bytes) and the name of the members are described in the tables 2 and 3:

line

column

latitude

longitude

zenith_angle

emissivity[nchannels]

land_sea_type

topo

bt_satellite[n_channels]

bias_offset[n_channels]

bias_scale[_nchannels]

bt_rms_threshold

max_residual

max_iterations

emissivity_IR97

bt_satellite_IR97

bias_offset_IR97

bias_scale_IR97

Table 2: members in “inputs&ancillary” structure

bt_rttov[n_channels]

SKT

p_sfc

z_sfc

T_sfc

q_sfc

p[54]

T[54]

q[54]

O₃[54]

TPW

SHW

TOZ

Bt_rms

residual

iteration

qc_flag

Table 3: members in “iSHAI_profile” structure

The units of q profiles at the 54 RTTOV pressure levels are ppmv. Same for Ozone profile.

The position on region coordinates (i=element, j=line) are included in every record together with ancillary parameter (longitude, latitude, etc) in order to geolocate the FOR.

For BT_SATELLITE and BT_RTTOV arrays have dimension n_channels and the order is from lower to higher wavelength.

For synthetic or simulated brightness temperature binary files, RTTOV BTs for channels are stored in the BT_RTTOV arrays and have been obtained after applying RTTOV forward to the stored retrieved profiles.

The fields on the GEO-iSHAI binary files have been unified for HYB and P mode and just differ in the name of the binary file because the GEO-iSHAI execution mode has been included in the name.

Once any of the keywords are activated in the configuration file, the GEO-iSHAI optional binary files will be written in the $SAFNWC/tmp directory in the next execution of GEO-iSHAI. The GEO-iSHAI optional binary files are always written at $SAFNWC/tmp directory.

END PROFILE at clear FOR (SAVE_iSHAI_in_BINARY_file 1)

Fixing SAVE_iSHAI_in_BINARY_file to 1, the final retrieved profiles (T, q, ozone profile at the 54 RTTOV levels) and other parameters are written only at clear FOR or pixels.

These are the profiles used to calculate TPW, LPW, stability indices and TOZ together with the difference with background NWP profiles. The name of END PROFILE at clear air FOR optional binary file is:

iSHAI_mode_END_____CLEAR_nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin

where:

CLEAR indicates only at clear air FOR
mode processing mode P or HYB
nameregion name of region
Clinc_colc_Ssizl_sizc position of region centre (Clinc_colc) and size of region (Ssizl_sizc)
YYYY-MM-DDThh:mm:00Z is the date (YYYY-MM-DD) and hour (hh:mm:00Z) of SEVIRI image

Example:

iSHAI_HYB_END_____CLEAR_LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin

In this case the binary file is a 1D array [number_clear_FOR] containing values of the structures written on the clear air FOR or pixels where the GEO-iSHAI algorithm has been really executed.

The size of this file is (size_of_record x number_clear_FOR).

END PROFILE at ALL pixels (SAVE_iSHAI_in_BINARY_file 2)

Fixing SAVE_iSHAI_in_BINARY_file to 2, the final retrieved profiles (T, q, ozone profile at the 54 RTTOV levels) and other parameters are written for all the pixels. At clear FOR or pixels these profiles are the profiles used to calculate GEO-iSHAI parameters and for cloudy or not processed pixels dummy structures filled all members with missing value (-9999.0) are written.

The name of END PROFILE at all pixels optional binary file is:

iSHAI_mode_END_____nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin

where:

Mode processing mode P or HYB
Nameregion name of region
Clinc_colc_Ssizl_sizc position of region centre (linc, colc are the line and column coordinates of the centre of the region (0-based)) and size of region (Ssizl_sizc are the size of the region (number of lines and number of columns)).
YYYY-MM-DDThh:mm:00Z is the date (YYYY-MM-DD) and hour (hh:mm:00Z) of SEVIRI image

Example:

iSHAI_HYB_END_____LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin

In this example LargeIberica is a region of 665 lines x 511 columns centred in (558, 1758) by this reason the name of binary file contains C1748_0558_S0665_0511

In this case the binary file is a 2D array [ncols, nlines] containing values of the structures written on all pixels. These binary files are adequate for small to medium regions when processing in 1x1 FOR and in applications as vertical cross sections, 3D visualizations, etc.

Note: users must take into account the huge size of the binary files that could be generated and the very expensive CPU time of the pixel by pixel processing for large regions.

The size of this file is (size_of_record x ncols x nlines)

Background NWP and FG profiles at clear FOR (SAVE_PROFs_in_BINARY_files 1)

Fixing SAVE_PROFs_in_BINARY_files to 1, the background NWP and after FG non-linear regressions profiles (T, q, ozone profile at the 54 RTTOV levels) and other parameters are written only at clear FOR or pixels. These are the profiles used to calculate TPW, LPW, stability indices and TOZ together of the difference with the end GEO-iSHAI profiles. The name of background nwp and FG at clear air FOR optional binary files are respectively:

iSHAI_mode_nwp_____CLEAR_nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin

iSHAI_mode_FG______CLEAR_nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin

where:

CLEAR indicates only at clear air FOR
Mode processing mode P or HYB
Nameregion name of region
Clinc_colc_Ssizl_sizc position of region centre (Clinc_colc) and size of region (Ssizl_sizc)
YYYY-MM-DDThh:mm:00Z is the date (YYYY-MM-DD) and hour (hh:mm:00Z) of SEVIRI image

Examples:

iSHAI_HYB_nwp_____CLEAR_LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin

iSHAI_HYB_FG______CLEAR_LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin

In this case the binary files are a 1D array [number_clear_for] containing values of the structures written on the clear air FOR or pixels where the GEO-iSHAI algorithm has been really executed.

The size of this file is (size_of_record x number_clear_FOR).

Background NWP and FG profiles at ALL pixels (SAVE_PROFs_in_BINARY_files 2)

Fixing SAVE_PROFs_in_BINARY_files to 2, the background NWP and after FG non-linear regressions profiles (T, q, ozone profile at the 54 RTTOV levels) and other parameters are written for all the pixels. These are the profiles used to calculate GEO-iSHAI parameters at clear FOR or pixels and dummy structures filled all members with missing value (-9999.0) for cloudy or not processed pixels.

The name of background nwp and FG at all pixels optional binary files are respectively:

iSHAI_mode_nwp_____nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin

iSHAI_mode_FG______nameregion_Clinc_colc_Ssizl_sizc_YYYY-MM-DDThh:mm:00Z.bin

where:

Mode processing mode P or HYB
Nameregion name of region
Clinc_colc_Ssizl_sizc position of region centre (Clinc_colc) and size of region (Ssizl_sizc)
YYYY-MM-DDThh:mm:00Z is the date (YYYY-MM-DD) and hour (hh:mm:00Z) of SEVIRI image

Examples:

iSHAI_HYB_nwp_____LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin

iSHAI_HYB_FG______LargeIberica_C1748_0558_S0665_0511_2015-07-15T00:15:00Z.bin

In this case the binary file are a 2D array [ncols, nlines] containing values of the structures written on all pixels. It is more useful when processing in 1x1 FOR.

The size of this file is (size_of_record x ncols x nlines)

Note: users must take into account the huge size of the binary files that could be generated and the very expensive CPU time of the pixel by pixel processing for large regions.

Conversion from binary files to netCDF

The binary files are not easy to manage and since 2010 some IDL procedures have been developed to convert the iSHAI binary files to netCDF format.

In order to facilitate the use of the binary files, tools to convert the GEO-iSHAI binary files to netCDF format has being developed adapting the IDL prototypes available from former PGE13 SPhR ones. These conversion tools from GEO-iSHAI binary files to netCDF format are being generated at the best effort basis from the IDL prototypes. One possibility it use the GNU version of IDL called GDL; the IDL prototypes will tested and adapted to be compatible with GDL if possible. The other possibility it to develop FORTRAN, C or Python versions; these could be written using the API netCDF library and the IDL versions as basis.

The IDL (or GDL) procedures are now supported as best effort basis (on request to mmartinezr@aemet.es currently). At the present time, the generation of web page in the NWC SAF web server for support of the conversion tool from binary to netCDF format is in progress.

There are a variety of software packages that can read the netCDF format and can be used by the users to a wider and deeper exploitation of the GEO-iSHAI binary files.

In our case, the structure of the netCDF files have been designed to be compatible with several meteorological tools like McIDAS-V and IDV (they are freely available on their respective web pages) to produce 2D and 3D displays interactively. Also the netCDF files can be read and used directly in application with Python, IDL, MATLAB, etc.

Exploitation with 2D and 3D interactive tools

There are a great variety of software packages that can also read the netCDF format and can be used by the users to a wider and deeper exploitation of the GEO-iSHAI binary files. Among all software packages IDL, MATLAB, GDAL, McIDAS-V, IDV, Python netCDF reader, etc. could be used for easy management of the GEO-iSHAI binary files and/or netCDF for validation and/or visualization purposes.

In order to show examples of the important applications for monitoring of change using these optional binary files, tools freely available as IDV, McIDAS-V could be used as example to get all kind of interactive 2D and 3D visualization. See the vertical cross sections on the case study included in the visualization examples.In the link several examples of the use of vertical cross sections for nowcasting for a case study is presented. For more examples of the possibilities you can see also the Martinez presentations and links in the iSHAI references page.