{"id":24951,"date":"2022-03-28T12:53:31","date_gmt":"2022-03-28T11:53:31","guid":{"rendered":"https:\/\/www.ch.imperial.ac.uk\/rzepa\/blog\/?p=24951"},"modified":"2022-03-29T21:07:59","modified_gmt":"2022-03-29T20:07:59","slug":"raw-data-and-the-evolution-of-crystallographic-fair-data-journals-processed-and-raw-structure-data","status":"publish","type":"post","link":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951","title":{"rendered":"Raw data and the evolution of crystallographic FAIR data. Journals, processed and raw structure data."},"content":{"rendered":"
\n

In my previous post on the topic<\/a>,\u00a0I introduced the concept that data can come in several forms, most commonly as “raw” or primary data and as a “processed” version of this data that has added value. In crystallography, the chemist is interested in this processed version, carried by a CIF file. However on rare occasions when a query arises about the processed component, this can in principle at least be resolved by taking a look at the original raw data, expressed as diffraction images.\u00a0I established with much appreciated help from CCDC that since 2016, around 65 datasets in the CSD (Cambridge structural database) have appeared with such associated raw data. The problem is easily reconciling the two sets of data (the raw data is not stored on CSD) and one way of doing this is via<\/em> the metadata associated with the datasets. In turn, if this metadata is suitably registered, one can query the metadata store for such associations, as was illustrated in the previous post on the topic. Here\u00a0I explore the metadata records for five of these 65 sets to find out their properties, selected to illustrate the five data repositories thus far that host such data for compounds in the CSD database.<\/p>\n\n\n\n\n\n\n\n\n
Raw data
\nrepository<\/th>\n		Raw Data
\nDOI<\/th>\n		Raw data
\n\u2192CSD?<\/th>\n		CSD\u2192
\nRaw data?<\/th>\n		\u21d0Journal\u21d2<\/th>\n<\/tr>\n
Zenodo<\/small><\/td>\n10.5281\/zenodo.4271549<\/small><\/a><\/td>\nNo<\/a><\/td>\nNo<\/a><\/td>\n10.1039\/C6RA28567H<\/small><\/a><\/td>\n<\/tr>\n
Imperial College research data repository<\/small><\/td>\n10.14469\/hpc\/2298<\/small><\/a><\/td>\nYes<\/a><\/td>\nYes<\/a><\/td>\n10.1021\/acsomega.7b00482<\/small><\/a><\/td>\n<\/tr>\n
RepoD, a Harvard Dataverse instance<\/small><\/td>\n10.18150\/repod.6628285<\/small><\/a><\/td>\nNo<\/a><\/td>\nNo<\/a><\/td>\n10.1021\/acs.cgd.0c01252<\/small><\/a><\/td>\n<\/tr>\n
Cambridge university repository<\/small><\/td>\n10.17863\/CAM.21968<\/small><\/a><\/td>\nNo<\/a><\/td>\nNo<\/a><\/td>\n10.1016\/j.inoche.2018.08.024<\/small><\/a><\/td>\n<\/tr>\n
Isis neutron and muon source data journal<\/small><\/td>\n10.5286\/ISIS.E.RB1620465<\/small><\/a><\/td>\nNo<\/a><\/td>\nNo<\/a><\/td>\n10.1039\/D0CC02418J<\/small><\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Ideally, one is looking for bidirectional links between the data as expressed in the metadata and in both directions. As you can see from the above, these links are present in only one of the five sets. More common is that both the raw and the processed data will contain links to the journal article where the data is discussed. Very much less commonly are there links from the journal article to the raw data, although such links are slightly more likely to exist from the journal to the processed data. If you click on the link in any of the last three columns, a copy of the metadata will download for you to inspect. There you can verify if the assertions made above are correct.\u00a0<\/p>\n

What the metadata records demonstrate above is a very small scale so-called PID graph (DOI: [1]<\/a><\/span> 10.5438\/jwvf-8a66<\/a>)\u00a0where each DOI is a node in that graph and if a connection exists, it is shown by a line connecting the nodes.\u00a0The PID graph can be extended to include a third type of node, the journal article and then it starts to get interesting!\u00a0I will investigate if I can generate the PID graph for the above, although be prepared, it will not (yet) contain very many lines between nodes!<\/p>\n

References<\/h2>\n
  1. M. Fenner, and A. Aryani, \"Introducing the PID Graph\", 2019. https:\/\/doi.org\/10.5438\/jwvf-8a66<\/a>\n\n<\/li>\n<\/ol>\n\n<\/div> ","protected":false},"excerpt":{"rendered":"

    In my previous post on the topic,\u00a0I introduced the concept that data can come in several forms, most commonly as “raw” or primary data and as a “processed” version of this data that has added value. In crystallography, the chemist is interested in this processed version, carried by a CIF file. However on rare occasions […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":5,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[2],"tags":[],"ppma_author":[2661],"class_list":["post-24951","post","type-post","status-publish","format-standard","hentry","category-chemical-it"],"yoast_head":"\nRaw data and the evolution of crystallographic FAIR data. Journals, processed and raw structure data. - Henry Rzepa's Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Raw data and the evolution of crystallographic FAIR data. Journals, processed and raw structure data. - Henry Rzepa's Blog\" \/>\n<meta property=\"og:description\" content=\"In my previous post on the topic,\u00a0I introduced the concept that data can come in several forms, most commonly as “raw” or primary data and as a “processed” version of this data that has added value. In crystallography, the chemist is interested in this processed version, carried by a CIF file. However on rare occasions […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951\" \/>\n<meta property=\"og:site_name\" content=\"Henry Rzepa's Blog\" \/>\n<meta property=\"article:published_time\" content=\"2022-03-28T11:53:31+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-03-29T20:07:59+00:00\" \/>\n<meta name=\"author\" content=\"Henry Rzepa\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Henry Rzepa\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Raw data and the evolution of crystallographic FAIR data. Journals, processed and raw structure data. - Henry Rzepa's Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951","og_locale":"en_GB","og_type":"article","og_title":"Raw data and the evolution of crystallographic FAIR data. Journals, processed and raw structure data. - Henry Rzepa's Blog","og_description":"In my previous post on the topic,\u00a0I introduced the concept that data can come in several forms, most commonly as “raw” or primary data and as a “processed” version of this data that has added value. In crystallography, the chemist is interested in this processed version, carried by a CIF file. However on rare occasions […]","og_url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951","og_site_name":"Henry Rzepa's Blog","article_published_time":"2022-03-28T11:53:31+00:00","article_modified_time":"2022-03-29T20:07:59+00:00","author":"Henry Rzepa","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Henry Rzepa","Estimated reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951#article","isPartOf":{"@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951"},"author":{"name":"Henry Rzepa","@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/#\/schema\/person\/2b40f7b9c872a4dc1547e040a11b6281"},"headline":"Raw data and the evolution of crystallographic FAIR data. Journals, processed and raw structure data.","datePublished":"2022-03-28T11:53:31+00:00","dateModified":"2022-03-29T20:07:59+00:00","mainEntityOfPage":{"@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951"},"wordCount":502,"commentCount":2,"articleSection":["Chemical IT"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951","url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951","name":"Raw data and the evolution of crystallographic FAIR data. Journals, processed and raw structure data. - Henry Rzepa's Blog","isPartOf":{"@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/#website"},"datePublished":"2022-03-28T11:53:31+00:00","dateModified":"2022-03-29T20:07:59+00:00","author":{"@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/#\/schema\/person\/2b40f7b9c872a4dc1547e040a11b6281"},"breadcrumb":{"@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24951#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog"},{"@type":"ListItem","position":2,"name":"Raw data and the evolution of crystallographic FAIR data. Journals, processed and raw structure data."}]},{"@type":"WebSite","@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/#website","url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/","name":"Henry Rzepa's Blog","description":"Chemistry with a twist","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/#\/schema\/person\/2b40f7b9c872a4dc1547e040a11b6281","name":"Henry Rzepa","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/secure.gravatar.com\/avatar\/897b6740f7f599bca7942cdf7d7914af5988937ae0e3869ab09aebb87f26a731?s=96&d=blank&r=g370be3a7397865e4fd161aefeb0a5a85","url":"https:\/\/secure.gravatar.com\/avatar\/897b6740f7f599bca7942cdf7d7914af5988937ae0e3869ab09aebb87f26a731?s=96&d=blank&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/897b6740f7f599bca7942cdf7d7914af5988937ae0e3869ab09aebb87f26a731?s=96&d=blank&r=g","caption":"Henry Rzepa"},"description":"Henry Rzepa is Emeritus Professor of Computational Chemistry at Imperial College London.","sameAs":["https:\/\/orcid.org\/0000-0002-8635-8390"],"url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?author=1"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pDef7-6ur","jetpack-related-posts":[{"id":19180,"url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=19180","url_meta":{"origin":24951,"position":0},"title":"FAIR data \u21cc Raw data.","author":"Henry Rzepa","date":"December 7, 2017","format":false,"excerpt":"FAIR data is increasingly accepted as a description of what research data should aspire to; Findable, Accessible, Inter-operable and Re-usable, with Context added by rich metadata (and also that it should be Open). But there are two sides to data, one of which is the raw data emerging from say\u2026","rel":"","context":"In "Chemical IT"","block_context":{"text":"Chemical IT","link":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?cat=2"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":24723,"url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=24723","url_meta":{"origin":24951,"position":1},"title":"Raw data: the evolution of FAIR data and crystallography.","author":"Henry Rzepa","date":"March 1, 2022","format":false,"excerpt":"Scientific data in chemistry has come a long way in the last few decades. Originally entangled into scientific articles in the form of tables of numbers or diagrams, it was (partially) disentangled into supporting information when journals became electronic in the late 1990s. The next phase was the introduction of\u2026","rel":"","context":"In "Chemical IT"","block_context":{"text":"Chemical IT","link":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?cat=2"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":18646,"url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=18646","url_meta":{"origin":24951,"position":2},"title":"Accessing (raw) chemical data: a peek into the CIF format.","author":"Henry Rzepa","date":"July 21, 2017","format":false,"excerpt":"There is much focus at the moment on how to ensure experimental replicability in e.g. the molecular sciences. An important aspect of that is having access to FAIR data; data which is findable, accessible, inter-operable and re-usable. One of the \"gold standards\" in chemistry is the data associated with crystal\u2026","rel":"","context":"In "Chemical IT"","block_context":{"text":"Chemical IT","link":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?cat=2"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":20342,"url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=20342","url_meta":{"origin":24951,"position":3},"title":"Open Access journal publishing debates – the elephant in the room?","author":"Henry Rzepa","date":"November 4, 2018","format":false,"excerpt":"For perhaps ten years now, the future of scientific publishing has been hotly debated. The traditional models are often thought to be badly broken, although convergence to a consensus of what a better model should be is not apparently close. But to my mind, much of this debate seems to\u2026","rel":"","context":"In "Chemical IT"","block_context":{"text":"Chemical IT","link":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?cat=2"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":18257,"url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=18257","url_meta":{"origin":24951,"position":4},"title":"The challenges in curating research data: one case study.","author":"Henry Rzepa","date":"April 28, 2017","format":false,"excerpt":"Research data (and its management) is rapidly emerging as a focal point for the development of research dissemination practices. An important aspect of ensuring that such data remains fit for purpose is identifying what curation activities need to be associated with it. Here I revisit one particular case study associated\u2026","rel":"","context":"In "Chemical IT"","block_context":{"text":"Chemical IT","link":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?cat=2"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.ch.ic.ac.uk\/rzepa\/blog\/wp-content\/uploads\/2017\/04\/077-1.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":16628,"url":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?p=16628","url_meta":{"origin":24951,"position":5},"title":"Managing (open) NMR data: a working example using Mpublish.","author":"Henry Rzepa","date":"August 1, 2016","format":false,"excerpt":"In March, I posted from\u00a0the ACS meeting in San Diego on the topic of Research data: Managing spectroscopy-NMR, and noted a talk by MestreLab Research on\u00a0how a tool called Mpublish in the forthcoming release of their NMR analysis software Mestrenova could help. With that release now out, the opportunity arose\u2026","rel":"","context":"In "Chemical IT"","block_context":{"text":"Chemical IT","link":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/?cat=2"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_likes_enabled":false,"authors":[{"term_id":2661,"user_id":1,"is_guest":0,"slug":"admin","display_name":"Henry Rzepa","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/897b6740f7f599bca7942cdf7d7914af5988937ae0e3869ab09aebb87f26a731?s=96&d=blank&r=g","0":null,"1":"","2":"","3":"","4":"","5":"","6":"","7":"","8":""}],"_links":{"self":[{"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=\/wp\/v2\/posts\/24951","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=24951"}],"version-history":[{"count":21,"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=\/wp\/v2\/posts\/24951\/revisions"}],"predecessor-version":[{"id":24973,"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=\/wp\/v2\/posts\/24951\/revisions\/24973"}],"wp:attachment":[{"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=24951"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=24951"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=24951"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.ch.ic.ac.uk\/rzepa\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fppma_author&post=24951"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}