All four sit in one file of Python I committed myself and never touched again: one commit, the eleventh of the 32 in that repository. Each is a couple of lines that read fine and threw away a whole category of triple, and none of them raised an error, because a filter that matches nothing is indistinguishable from a filter that has nothing to match.
i
The essentials chunk looks for a syntax N-Triples never writes
utils/data-processing/prepare_taxon_rdf_triple_store.py:393–397
for line in lines:
if any(pattern in line for pattern in ['rdf:type', 'rdfs:label', 'dwc:scientificName']):
essentials.append(line)
All three of those are CURIEs, and the serialiser writes N-Triples, which spells every predicate out in full as <http://www.w3.org/2000/01/rdf-schema#label>, so the substring 'rdfs:label' occurs in exactly none of the 2,529,435 generated lines. The chunk I wrote to hold the most important data is on disk as a 45-byte gzip stream holding zero triples, and its manifest entry says triple_count 0.
- essentials.ntriples.gz
- 45 bytes
- triples in it
- 0
ii
The alphabetical filter and the label generator disagree about what a taxon is called
prepare_taxon_rdf_triple_store.py:408 and 182–191, with data/darwin-core/output-clean/taxon.tab
if 'urn:lsid:plantsforhealth.kew.org:taxon:' in line:
…
# but the label was minted against taxonID, which is an IPNI LSID:
taxon_uri = self.create_safe_uri(taxon_id) # urn:lsid:ipni.org:names:1000018-1
self.graph.add((taxon_uri, RDFS.label, Literal(row['scientificName'])))
self.graph.add((taxon_uri, DWC.scientificName, Literal(row['scientificName'])))
The taxon.tab in that tree has 54,207 rows, which is what the manifest reports processed, and 38,661 of them carry a scientificName, so the generator minted 38,661 rdfs:label triples and 38,661 dwc:scientificName triples. Every one of them hangs off an IPNI LSID. The alphabetical chunker keeps a line only if it contains the string urn:lsid:plantsforhealth.kew.org:taxon:, and the relationship chunker only dispatches on relationship words, so all of them fell through both filters. Not one label or scientific name is on disk, and the two rdf:type assertions per taxon went the same way.
- rdfs:label in the chunks
- 0
- dwc:scientificName in the chunks
- 0
iii
Two relationship chunks were never written, and a third holds the wrong thing
prepare_taxon_rdf_triple_store.py:426–439
if 'subClassOf' in line: chunks['taxonomic'].append(line)
elif 'has_part' in line: chunks['structural'].append(line)
elif 'contains' in line: chunks['chemical'].append(line)
elif 'has_role' in line: chunks['functional'].append(line)
elif 'hasVernacularName' in line: chunks['vernacular'].append(line)
has_part serialises as BFO_0000051 and has_role as RO_0000087, so relationships-structural and relationships-functional do not exist on disk and RO_0000087 occurs zero times in the whole bundle. The 'contains' branch fares worse than nothing: it matches the English word inside a description literal, so relationships-chemical is 96,200,279 bytes of sentences like "Artemisia annua L. contains CAMPHENE-HYDRATE" and holds not one RO_0001019 edge. The real contains edges survived only because they happen to mention a taxon URI and so landed in an alphabetical chunk.
- relationship chunks the code names
- 5
- relationship chunks on disk
- 3
- obo:RO_0000087 shipped
- 0
iv
The vernacular names kept their labels in the graph and lost them on the way out
prepare_taxon_rdf_triple_store.py:281–295
vernacular_uri = self.create_safe_uri(vernacular_name_id)
self.graph.add((vernacular_uri, RDF.type, PFH.VernacularName))
self.graph.add((vernacular_uri, RDFS.label, Literal(vernacular_name)))
self.graph.add((vernacular_uri, PFH.vernacularNameText, Literal(vernacular_name)))
…
if taxon_id in self.taxon_cache:
taxon_uri = self.taxon_cache[taxon_id]
self.graph.add((taxon_uri, PFH.hasVernacularName, vernacular_uri))
180,054 vernacular rows were processed and every one got an rdfs:label carrying the actual word. The subject of that label is a vernacularName URI, which matches neither filter, so none of those labels ever reached a chunk. What shipped is the pointing half of the pair: 266,555 hasVernacularName edges aimed at bare numbers. 180,017 of those hang off IPNI LSIDs that appear as a subject nowhere else in the bundle and as an object nowhere at all, which leaves them pointing out of the graph from a node you cannot reach. The remaining 86,538 do hang off taxon URIs, and for every one of the 8,562 taxa involved the number of ids is exactly the number of name strings sitting in that taxon's description literals. The two halves are the same size and nothing joins them.
- pfh:hasVernacularName edges
- 266,555
- off an orphan IPNI subject
- 180,017
- ipni URIs used as an object
- 0
One more, and this one is not mine
1,412 taxa on disk have more than one named part, and 1,262 of them (89.4%) assert an identical compound set on every single part. Camellia sinensis (L.) Kuntze carries 15 parts with compounds recorded against them, from anther to tissue culture, and every one is said to hold the same 641. Papaver somniferum L. is 17 parts by 247, Curcuma longa L. six by 180. It is a cross product, and it arrives that way from upstream rather than from anything my chunker did: in the Darwin Core export sitting in that tree, 999 taxa have more than one part with compounds recorded against it and 794 of them repeat one identical set across every part. The 150 on disk that do vary are the interesting ones.
- Malus domestica (Suckow) Borkh. 22 parts, 3 to 470 compounds
- Apium graveolens L. 12 parts, 3 to 469 compounds
- Punica granatum L. 18 parts, 3 to 265 compounds
- Salvia officinalis L. 14 parts, 1 to 255 compounds
- Citrullus lanatus (Thunb.) Matsum. & Nakai 15 parts, 7 to 217 compounds
- Thymus vulgaris L. 6 parts, 2 to 187 compounds