- Table of contents
- Tutorial
- Using CDO from other languages
- Interpolation
- Plotting
Maybe davao or CDO is better like you said did you go there from the dating sites? Yeah teh girl im with now didn't have a job at the moment or degree she is basically just mooching off me. She is a decent girl in the sense she isn't a total slut with a big past. Free dating site, friendship and social discovery. Meet girls in Cdo online now. Cdo's best FREE dating site! 100% Free Online Dating for Cdo Singles at Mingle2.com. Our free personal ads are full of single women and men in Cdo looking for serious relationships, a little online flirtation, or new friends to go out with. Start meeting singles in Cdo today with our free online personals and free Cdo. 6,857 likes 1 talking about this.
The Climate Data Operators (CDO) software is a collection of many operators for standard processing of climate and forecast model data. The operators include simple statistical and arithmetic functions, data selection and subsampling tools, and spatial interpolation. CDO was developed to have the same set of processing functions for GRIB and NetCDF datasets in one package.
The Climate Data Interface [CDI] is used for the fast and file format independent access to GRIB and NetCDF datasets. The local MPI-MET data formats SERVICE, EXTRA and IEG are also supported.
There are some limitations for GRIB and NetCDF datasets.
A GRIB dataset has to be consistent, similar to NetCDF. That means all time steps need to have the same variables, and within a time step each variable may occur only once.
NetCDF datasets are only supported for the classic data model and arrays up to 4 dimensions. These dimensions should only be used by the horizontal and vertical grid and the time. The NetCDF attributes should follow the GDT, COARDS or CF Conventions.
Introduction¶
To display all command line options of cdo, type
and to get an overview of all operators with a short description, type
To get more information about an operator
The best way to get full information about the operators is to read the document [[https://code.mpimet.mpg.de/projects/cdo/embedded/index.html]]
First, before you start to work with your data you should have a closer look at it to know which variables are
stored, what the grid looks like, and not to forget to see which global, variable and dimension attributes are
defined.
With the 'cdo info' command you can see the timesteps, levels, minimum, maximum, averages and missing values. Type
and with ncdump from the NCO's all metadata and data contents can be displayed. To display the metadata of the file,
type
Basic Usage¶
- Display variables of a file
- Display number of timesteps of a file
- Display Information about the underlying grid:griddes does the following:
- File conversion with different file types: Copying whole data sets can be easily done with the copy operator.
With the '-f' switch, you can choose a new file type: Combine this with the '-z' options for changing to a higher compression ration: - Select variables from file: select variable tas select variables u10 and v10
- Select timesteps from file: e.g. select only the 3rd time step or select 3 timesteps or select a time range from 1 to 12 If you have a list of dates (e.g. format YYY-MM-DD) you can use the select operator:
- Select only data of the northern hemisphere (sub-region):
- Rearrange data from longitude 0 to 360 degrees to -180 to 180 degrees (latitude: -90 to 90 degrees):
- Invert the latitudes from north-south to south-north:
- Convert from K to degC when input file contains temperature values: and don't forget to change the variable (here tas) units, too. Combining operators:
- Set constant value to missing value: change data value -999.0 to be missing value or vice versa set missing value to constant value:
- Compute the monthly mean with respect to the number of days per month: don't forget to change the units attribute of the variable
- Delete February 29th:
Make variable modifications¶
The name of a variable and its attributes (metadata) can be modified with some CDO operators like chname, chcode, or setattribute.
To change the name of a variable from temp to t2m
To change the code number 98 to 179 and the code number 99 to 211:
To change the variable attribute units after some computations:
The setattribute operator accepts more than one attribute and it supports wildcards, e.g.:
Note!
The coordinate (dimension) variable names can't be changed by CDO but you can use NCO's ncrename to do it.
If you want to rename a coordinate variable so that it remains a coordinate variable, you must separately rename
both the dimension and the variable.
E.g. to rename the coordinate (dimension) variable names from ncl1, ncl2, ncl3 to time, lat, lon:
Combining Operators¶
All operators with one output stream can pipe the result directly to another operator. The operator must begin
with '-' in order to combine with others. This can improve the performance by:
- reducing unnecessary disk I/O: no intermediate files
- parallel processing: all operators in a chain work in parallel
- Simple combination:instead of
- Advanced Combination:
arbitrary list of input files cannot be combined with other operators:
The expr Operator¶
The expr opertor is propablely a rarely used but
so much more usefull tool. Its purpose is to compute complex math operation pointwise on arbitrary fields.
Let height.nc contain a 3d vertical coordinate variable z. For generating a pressure out of it, it is possible
to use expr like this:
To compute or select parts of your data and save it to a new file, e.g. create new variables tupper which contain
values greater equal 273.15 and tlower which contain values less than 273.15 of the variable tas:
To convert the variable tas to units degrees Celsius and compute the tupper and tlower variables:
The select and delete Operator¶
We already saw some of the selecting capabilities of CDO like seltimestep or selname. The select operator selects some fields from input files and write
it to an output file e.g. variable names, levels, dates or seasons. The delete
operator deletes some fields and write the result to an output file.
To chose a user defined season use the select operator with the parameter season where the given season is a comma separated
list of seasons (substring of DJFMAMJJA- SOND or ANN):
Select last timestep without knowing the total number of timesteps in infile:
Assume you have 3 input files. Each input file contains the same variables for a different time period. To select the
variable T,U and V on the levels 200, 500 and 850 from all 3 input files, use:
Delete first timestep in file:
Missing values¶
Sometimes you need to set or change the missing value of a variable, or change NaNs to missing value.
Set missing value to a constant value, e.g. -9999
Set constant value e.g. -999.9 to missing value
Set NaN to missing value and change the missing value to -9999.9
Autocompletion¶
In the contrib subdirectory of the official release are configuration files for using autocompletion with bash, zsh and
tcsh. For the current developement status they can be found here: source:/trunk/cdo/contrib.
For activation you have to let you shell read the corresponding file, e.g. for zsh: Same
work for bash and tcsh.
Using CDO from other languages¶
There are CDO bindings for Ruby and Python. Please have a look at cdo.{rb,py} for further information.
High performance processing¶
In the following example, a horizontal interpolation is performed on a large input file with many variables. For speed up
the input is splitted by variable name and each variable is processed in parallel. Computation is now limited by
IO-performance only, so this is most efficiently done using a ramdisk. In this
example, the parallelism is implemented via an additional Ruby module JobQueue
(github). It is a queue-like wrapper around the built-in multithreading. For python,
the modules multiprocessing or threading can be used.
Masking¶
Create a file with masked land:
or using the expr operator Create a file with masked ocean: or using the expr operator
Using the topo_land.nc file to mask the ocean part of your data:
data file: file_A.nc
mask file: topo_land.nc
Remap mask file to the data grid size of file test_A.nc:
Mask test_A.nc with topo_land_grid_A.nc to get the values of test_A.nc only over land:
Interpolation¶
Horizontal fields¶
To interpolate your data from an existing horizontal field to a finer or coarse grid or another grid type CDO provides
a set of operators starting with remap.
Available grid interpolation types:
To remap data to a global 1x1 degree grid:
To create a weights file once when you have to remap multiple files to e.g. Gaussian N32 grid. First, create a weights
file for the remapping only once:
Then do the remaping:
Tip:
To remap data to a different grid of another data file. Assume, your data file is called infile and you want to remap
it to the same grid as the file other_data.nc.
Use the other_data.nc as grid template to remap your infile to the same grid:
If you want to remap your data to an user-defined grid you have to create a file which describes the new grid (see CDO
doc https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-150001.3.2).
Dateinasia Cebu
Example grid description file gridfile.txt
Use the grid description file to remap the data to the new grid:
If you want to see the grid decription of your actual data file you can use the CDO operator griddes:
Vertical fields¶
Available grid interpolation operators:
To interpolate model levels to pressure levels, e.g.:
Time interpolation¶
Available time interpolation operators:
To interpolate 6-hourly data to 1-hourly data:
or more accurate
Interpolate time by number of timesteps from one timestep to the next:
Plotting¶
In order to rapidly generate high quality pictures from the data obtained from the existing CDO operators, the CDO
has been interfaced with Magics++ library. As a first step,some
new CDO plotting operators are created to cater to the most essential/ frequently used plotting features viz., graph,
contour, vector. These operators rely on the Magics++ and generate output files in the various formats supported by Magics++.
Magics++ is the latest generation of the ECMWF's Meteorological plotting software MAGICS.Magics++ supports the plotting
of contours, wind fields, observations, satellite images, symbols, text,axis and graphs (including box plots).Data fields
to be plotted may be presented in various formats, for instance GRIB 1 and 2 code data, gaussian grid, regularly spaced
grid and fitted data,BUFR and NetCDF format or retrieved from an ODB database.The produced meteorological plots can be
saved in various formats, such as PostScript, EPS, PDF, GIF, PNG and SVG.
Magics++ provides a vast number of parameters to control the attributes of various plotting features. Keeping in view,
the usability of CDO users, currently only a few of these parameters are supported and accessible to the CDO users as
command line arguments for the respective operators.The users are requested to refer to the
Magics++ library for detailed description of the various parameters
available for the various features.
The description of the new plotting operators and the various arguments that can be passed for these operators is provided
in the subsequent sections. To explore these operators, the CDO should be build with '--with-magics' and '--with-libxml2'
config options.
graph¶
Usage
Dateinasia Online Dating
| device | Output File format can be 'PS','EPS','PDF','PNG','GIF','JPEG','SVG','KML' . Default output format is 'PS' |
| ymin | Minimum value of the y-axis data |
| ymax | Maximum value of the y-axis data |
| stat | 'TRUE' or 'FALSE' , to switch on the mean computation. Default is 'FALSE'. Will be overridden to 'FALSE' if input files have unequal number of time steps or different start/end times. |
| sigma | Standard deviation value for generating shaded back ground around the mean value.To be used in conjunction with 'stat='TRUE' |
| obsv | To indicate if the input files have an observation data, by setting to 'TRUE'.Default value is 'FALSE'. The observation data should be the first file in the input file list. The observation data is always plotted in black colour. |
Sample Images
Dateinasia Cdo 2
contour¶
Usage
| device | Output File format can be 'PS','EPS','PDF','PNG','GIF','GIF_ANIMATION','JPEG','SVG','KML' . Default output format is 'PS'. |
| min | Minimum value of the data for generating the contour plot |
| max | Maximum value of the data for generating the contour plot |
| count | Number of contour levels |
| interval | Interval in data units between two bands lines |
| list | List of levels to be plotted |
| RGB | To indicate, if the input colour is in RGB format |
| colour | Colour for drawng the contours, can be either standard name or in RGB format |
| thickness | Thickness of the contour line |
| style | Line Style can be 'SOLID','DASH','DOT','CHAIN_DASH','CHAIN_DOT' |
| step_freq | Frequency of time steps to be considered for making the animation( device = gif_animation). Default value is '1' (all time steps). Will be ignored if input file has multiple variables |
| file_split | To split the output file for each variable, if input has multiple variables.Values can be 'TRUE' or 'FALSE'. Default value is 'FALSE'. Valid only for 'PS' format. |
shaded¶
Usage
| device | Output File format can be 'PS','EPS','PDF','PNG','GIF','GIF_ANIMATION','JPEG','SVG','KML' . Default output format is 'PS' |
| min | Minimum value of the data for generating the grid fill plot |
| max | Maximum value of the data for generating the grid fill plot |
| count | Number of colour bands |
| interval | Interval in data units between two bands lines |
| list | List of levels to be plotted |
| RGB | To indicate, if the input colour (i.e,colour_min, colour_max) is in RGB format |
| colour_min | Colour for the Minimum colour band,can be either standard name or in RGB format |
| colour_max | Colour for the Minimum colour band,can be either standard name or in RGB format |
| colour_triad | Direction of colour sequencing for shading 'CW' or 'ACW' , to denote 'clockwise' and 'anticlockwise' respectively. To be used in conjunction with 'colour_min', 'colour_max' options. Default is 'anticlockwise' |
| colour_table | File with user specified colours with the format as ( example file for 6 colours in RGB format )(can be either standard name or in RGB format.) |
| step_freq | Frequency of time steps to be considered for making the animation( device = gif_animation). Default value is '1' (all time steps). Will be ignored if input file has multiple variables |
| file_split | To split the output file for each variable, if input has multiple variables.Values can be 'TRUE' or 'FALSE'. Default value is 'FALSE'. Valid only for 'PS' format. |
grfill¶
Usage
| device | Output File format can be 'PS','EPS','PDF','PNG','GIF','GIF_ANIMATION','JPEG','SVG','KML' . Default output format is 'PS' |
| min | Minimum value of the data for generating the grid fill plot |
| max | Maximum value of the data for generating the grid fill plot |
| count | Number of colour bands |
| interval | Interval in data units between two bands lines |
| list | List of levels to be plotted |
| RGB | To indicate, if the input colour (i.e,colour_min, colour_max) is in RGB format |
| colour_min | Colour for the Minimum colour band,can be either standard name or in RGB format |
| colour_max | Colour for the Minimum colour band,can be either standard name or in RGB format |
| colour_triad | Direction of colour sequencing for shading 'CW' or 'ACW' , to denote 'clockwise' and 'anticlockwise' respectively. To be used in conjunction with 'colour_min', 'colour_max' options. Default is 'anticlockwise' |
| resolution | Number of cells per cm for CELL shading |
| colour_table | File with user specified colours with the format as ( example file for 6 colours in RGB format )( can be either standard name or in RGB format ) |
| step_freq | Frequency of time steps to be considered for making the animation( device = gif_animation). Default value is '1' (all time steps). Will be ignored if input file has multiple variables |
| file_split | To split the output file for each variable, if input has multiple variables.Values can be 'TRUE' or 'FALSE'. Default value is 'FALSE'. Valid only for 'PS' format. |
vector¶
Usage
Important: the vector variables have to be 'u' and 'v'. To change the variable names e.g. from 'u10' and 'v10' use -chname,u10,u,v10,v before creating the vector plot like:
| device | Output File format can be 'PS','EPS','PDF','PNG','GIF','GIF_ANIMATION','JPEG','SVG','KML' . Default output format is 'PS' |
| thin_fac | Controls the actual number of wind arrows or flags plotted. See Magics++ library for explanation |
| unit_vec | Wind speed in m/s represented by a unit vector ( 1.0 cm ) |
| step_freq | Frequency of time steps to be considered for making the animation( device = gif_animation). Default value is '1' (all time steps). Will be ignored if input file has multiple variables |
NOTE: The valid standard name strings for 'colour' are
'red', 'green', 'blue', 'yellow', 'cyan', 'magenta', 'black', 'avocado',
'beige', 'brick', 'brown', 'burgundy','charcoal', 'chestnut', 'coral', 'cream',
'evergreen', 'gold', 'grey', 'khaki', 'kellygreen', 'lavender',
'mustard', 'navy', 'ochre', 'olive','peach', 'pink', 'rose', 'rust', 'sky',
'tan', 'tangerine','turquoise','violet', 'reddishpurple',
'purplered', 'purplishred','orangishred', 'redorange', 'reddishorange',
'orange', 'yellowishorange','orangeyellow', 'orangishyellow',
'greenishyellow', 'yellowgreen','yellowishgreen', 'bluishgreen',
'bluegreen', 'greenishblue','purplishblue', 'bluepurple',
'bluishpurple', 'purple', 'white'
Tips and tricks for high resolution data¶
High resolution data sets have a couple of extra properties, which make them a little harder to work with
- Input files are usually huge - quite obvious, I know, but it's good to keep that in mind if things take much longer than usual. Computing a global mean value of 80 million grid points just takes longer than on 80000, so be patient!
- The large file size implies an even larger influence of IO on the total performance. Early data reduction/selection is a must as well as avoidance of temporary file creation. It can be worth checking the actual reading/writing performance of your system in order to estimate which worklow works best: write interim results of mediate or large size and base next computations in them or work with complex CDO calls to write down very small result files.
- File size can be misleading though: NetCDF and GRIB2 have very effective compression algorisms built-in (zip-compressed nc4, aec/szip compressed grb2). The downside is that in both cases decompression is slow. Especially with large horizontal fields the time for decompressing supersedes the saved read-in time compared to uncompressed data. These compressions are essentially made for saving storage space, but not for extensive work with the data.
- Due to the fact, that the coordinates themselves are big fields, they are usually not attached to high resolution data files to save disk space. Hence they have to be attached on-the-fly for any of operation that needs them, e.g.
sellonlatbox,fldmean, ... - Don't hesitate to selection regions on unstructured grids with the usual
sellonlatboxoperator: Originally designed for regular grids, it now (cdo-1.9.5) uses very fast search algorithms for any kind of grids. - Some words on interpolation: Consider pre-computing interpolations weight even if you have multiple regions of interest. The weight generation itself is OpenMP-parallelized and scales well, the application of these weight be parallelized using multiple processes
Lets go through some examples step by step
Dateinasia Online
remapping
selection
any ideas?
CDO option --worker¶
Dateinasia Login
The CDO option '--worker <N>' set the number of workers to asynchronously decode/decompress <N> GRIB records.
This can improve the performance of reading compressed GRIB files. The speedup and memory usage depends on
the size of the grid, the number of records per timestep and the used CDO operators.
The option --worker is available since CDO release 1.9.7.
Here is an example of 'cdo info' on an AEC compressed GRIB2 file containing ICON R2B9 data with one 3D variable
on 77 levels and 8 timesteps:
The speedup with 4 workers is 3 and it doesn't scale with more than 8 workers.
The I/O is synchronized after each timestep. That means the worker pipeline needs to be flushed at the end
of a timestep and refilled at the beginning of the next timestep. Therefor the number of records per timestep
must be much higher than the number of worker in order to improve the performance.
Be careful when using any selection operator in combination with other CDO operators.
This will start all worker with decompressing the next records also if they are not selected.
This could slow down the performance of the processing.
| ifs2icon_skel.rb(1.14 KB)ifs2icon_skel.rb |
| teststat.jpg(70.1 KB)teststat.jpg |
| testall.jpg(74.1 KB)testall.jpg |
| cdo_worker.jpg(83.2 KB)cdo_worker.jpg |