The identifier scan and de-identification

Every dataset you upload is scanned for columns that look like they hold identifiers, such as names, record numbers, contact details or exact dates. This page explains what the scan looks for, what you can decide for each flagged column, and what the result is and is not. What the AI receives is covered separately in What we send to the AI.

In short

• The scan checks each column against the HIPAA Safe Harbor identifier categories, using the column name, the shape of its values and, for free text, the text itself.
• Detection is a careful heuristic, not a verdict. Unusual formats can be missed, so review your columns yourself too.
• In a standard workspace, an upload with a column the scan rates as a high-risk identifier is refused before anything is stored. Other flagged columns are kept and left out of what the AI is told about your dataset.
• You decide what happens to each flagged column. Applying your choices writes a new version of the dataset; the current version is kept.
• Celeus treats and labels a dataset with shifted dates as a Limited Data Set, not de-identified data. Whether it legally qualifies depends on the rest of the dataset.
• The result is not a legal determination of whether your dataset meets a de-identification standard.

What the scan looks for

The scan checks for these kinds of identifier:

• names
• geographic detail smaller than a state, such as an address, city or ZIP code
• dates finer than a year
• very high ages
• phone and fax numbers
• email addresses
• Social Security numbers
• medical record, account and health-plan numbers
• national identity numbers, such as the Turkish TC Kimlik No
• certificate and license numbers
• vehicle identifiers
• device identifiers and serial numbers
• web URLs
• IP addresses
• biometric identifiers
• full-face photographs and comparable images
• other unique identifiers and free text

It uses three kinds of evidence:

• The column name. A name such as “surname”, “dob” or “mrn” suggests what the column holds. Common English, Turkish and German words are recognised.
• The shape of the values. Values formatted like a phone number, an email address or a Social Security number are recognised from the values alone, including some phone and Social Security numbers stored without punctuation. Some shapes count only together with a matching column name.
• The text itself. For free-text columns, Celeus also scans the text content for identifiers.

Some kinds of identifier, such as vehicle identifiers, device serial numbers, biometric identifiers and photographs, are recognised mainly from the column name, so name your columns descriptively and review them yourself. You can also mark any column yourself, and the scan then treats it as a direct identifier.

The scan reads the first values of each column (a limited number). It reports column names, categories and counts, never the matched values themselves.

What happens at upload

In a standard workspace, an upload with a column the scan rates as a high-risk identifier is refused before anything is stored, and the message names the columns. If the free-text part of the scan cannot run, the upload is refused too, rather than accepted unchecked.

Columns flagged at a lower level are kept, marked as a PHI risk until you review them, and left out of what the AI is told about your dataset. You can include a flagged column again after review; see What we send to the AI.

A workspace owner or admin can instead choose to store identified data. That takes a second factor and accepting an attestation. In such a workspace, analysis and the AI features stay blocked on a dataset stored with identifiers until every flagged column has been de-identified or reviewed, as described below.

What you decide for each column

In Review de-identification, each flagged column gets an action. Celeus proposes one from what the scan found, and you can change it. Only the actions that fit the column are offered:

• Drop column removes the column.
• Pseudonymize replaces every value with a code. See Coded stand-ins and the linkage key.
• Keep year only, for a date column, replaces each date with its year. A value that cannot be read as a date becomes empty, and the de-identification log counts how many did.
• Shift all dates by one offset, for a date column. See Shifting dates.
• Cap ages over 89 at 90, for an age column, sets every age above 89 to 90.
• Keep first 3 ZIP digits, offered only for a column whose name and values look like ZIP codes, keeps the first three digits; values that are not ZIP-shaped become empty. Prefixes on Celeus’s built-in list of areas with 20,000 people or fewer (derived from 2000 Census data) are set to 000, because a three-digit prefix alone can identify such an area.
• Keep unchanged leaves the column exactly as uploaded. If the column really is an identifier, the dataset stays identifiable.
• Reviewed - not an identifier is for a column the scan got wrong. Nothing is removed, masked or changed: you attest that the column does not identify anyone, and that attestation is recorded against your account and the column. The column’s values are then used in analysis as uploaded.

Apply de-identification applies your choices. Not now leaves the dataset as it is, and that choice is recorded; you can de-identify it later.

When a block appears while you are working

If you try to run an analysis on a dataset that is blocked, the warning offers Review de-identification right there. It opens the same review over the page you are on, so you do not lose the analysis you were setting up, and once the block is cleared you can run it again from the same place. The analysis setup page also tells you beforehand, naming the columns that need review.

For each flagged column the review explains why it was flagged in plain words, with counts only, never the values. There are three ways to resolve one:

• It is not an identifier. A study code such as AV12, a gene or mutation name or a laboratory value can look like a phone number, a pager number or a record number to a scanner. Choose this only if the values are not built from a person’s initials, date of birth or record number. Choose Reviewed - not an identifier for the column and record the review. It is recorded under your name, the column and the date, and the data is not changed. To do this for several columns at once, use Mark several as not an identifier: you confirm once, and each column is still recorded separately. Columns flagged as names, email addresses or social security numbers, and columns you flagged yourself, are left out of that list and are reviewed one at a time.
• It is an identifier. Pseudonymize or drop the column. For dates, keep the year only or shift the dates.
• You are not sure. Leave the column flagged and ask whoever is responsible for data governance in your organisation. Analysis stays unavailable until the column is resolved.

If a pass still leaves columns flagged, the review lists them one by one with what to do next. A column whose free text could not be screened is not assumed to be clean: try again a little later, and ask your workspace admin if it keeps failing.

Coded stand-ins and the linkage key

Pseudonymize replaces each value with a code such as subj-001. Codes are assigned in order of first appearance, so the same value always gets the same code within that column, and a code is not derived from the value it replaces. The new version holds only the codes.

By default the mapping from values to codes is not kept, and your original file is the only way back. If you need to re-link codes to the original records later, tick Also download the linkage key when you apply. The linkage key is downloaded to your device. It is not stored on Celeus’s servers and is not put in the reproducible package, so keep it separately and under restricted access.

Shifting dates

Shifting dates moves every shifted date in the dataset by the same number of days, so intervals between dates stay exact. The offset is a whole number of days and never zero. It is drawn fresh each time. The offset is not recorded, so dates cannot be shifted back from the new version alone (the earlier version still exists).

Shifted dates are obscured, not removed, and HIPAA Safe Harbor does not allow that. Celeus therefore treats and labels a dataset with shifted dates as a Limited Data Set, not de-identified data. Whether it legally qualifies depends on the rest of the dataset. A Limited Data Set is still protected health information and may be disclosed for research, public health or health care operations only under a data use agreement. If any column is shifted, the whole output is treated as a Limited Data Set, and the de-identification log and any reproducible package say so. If you need Safe Harbor treatment of dates, keep the year only.

What happens to your original upload

De-identifying never overwrites a dataset. It writes a new version, and the current version is kept. Runs record which version they used. The earlier version, with its original values, stays in storage; de-identifying does not remove it.

When a reproducible package is built, columns the scan flagged that the analysis does not use are de-identified in the package’s copy of the data. See The reproducible package.

What this is not

The scan and the actions describe what Celeus checked and what it did to each column. They are not a legal opinion on whether a dataset meets a de-identification standard. A dataset can fail Safe Harbor for reasons a column scanner cannot see, for example an identifying detail typed into a notes column in an unusual format, or a combination of ordinary columns that points to one person. Whether your data may be uploaded, analysed or shared, and under which rules, is for you and your institution to decide.

For how Celeus protects data more broadly, see Security. Terms used on this page are defined in the glossary.