You are here: Home Programme Pre-conference workshops CDM workshop

CDM workshop

Motivations of the project

Since almost 20 years, the scientific community of neutrons and synchrotron facilities has been dreaming of using a common data format to be able to exchange experimental results and applications to analyse them. If using HDF5 as a physical container for data quickly raised a large consensus, the big issue is the standardisation of data organisation.

By introducing a new level of indirection for data access, the CommonDataModel (CDM) framework offers a solution and allows to split development efforts and responsibilities between institutes.

The CDM is made of a core API that access data through a data format plugins mechanism and scientific applications definitions (i.e. sets of logically organized keywords defined by scientists for each experimental technique). Using a innovative “mapping” system between applications definitions and physical data organizations, the CDM allows to develop data reduction applications regardless data files formats AND organisations.

Then each institute has to develop data access plugins for its own files formats along with the mapping between application definitions and its own data files organisation.

Thus, data reduction applications can be developed from a strictly scientific point of view and are natively able to process data coming from several institutes. 

 

Agenda of the workshop

  • Data file format : The end of the last decades debates ? : motivations of the project
  • Architecture of the CommonDataModel and its main concepts : data access, plugin, dictionary
  • Writing a CDM based data analysis application (code live demo)
  • Accessing data from the data analysis point of view using the simple Dictionary mechanism
  • Accessing data from the data analysis point of view using the advanced Dictionary mechanism
  • Writing a CDM plugin , to adapt your file format to CDM application (code live demo on a file format of one of the institute participating to the workshop)
  • Existing applications to access data through CommonDataModel :
    - for DataBrowsing (code live demo)

    - for Data Reduction : The FUSION framework
  • Accessing data from data analysis tools : MATLAB, python
  • Project organisation : Source code repository  and structure : How you can participate to the project
  • Conclusion and next steps of the project