DocumentationHow automated ingestion works
The stages of an automated ingestion workflow and the design choices that distinguish the available solutions.
Every automated ingestion workflow, whichever tool implements it, goes through the same stages. Understanding them helps you compare solutions and talk to your IT team.
1. Prepare
Before data are produced, the system needs to know where they belong. Typically, the operator selects a project, creates a dataset and starts an experiment (a single measurement run) in a web or desktop application. That record decides the target location in storage and the metadata form to fill in.
2. Acquire
The instrument and its vendor software work as usual and write files to a folder on the acquisition PC. Good solutions do not require changing the vendor software. At most, they ask the operator to save into a specific folder.
3. Transfer
Files move from the acquisition PC to managed storage. There are two basic approaches.
| Push (agent on the instrument PC) | Pull (central service fetches data) | |
|---|---|---|
| How | A small client app watches the data folder and uploads new or changed files | A central orchestrator, such as Apache Airflow, connects to a shared folder on the PC over SMB or SFTP and copies files |
| Instrument PC needs | The client installed, outbound HTTPS to storage | A shared folder and a service account, no extra software |
| Network | Simple: outgoing connections only | The PC can live in an isolated, deny-all network; only the orchestrator may reach it |
| Used by | DAREG Lab Client, Oneclient | DDRapp with DDRflow |
A third option avoids the transfer step entirely. Storage import means the instrument writes straight to a network storage that is already part of the managed system (see Onedata storage import).
Transfers should be incremental (only new or changed files), resumable, and should check that the received data match the source.
4. Register and describe
Each dataset gets a record in a registry with:
- Administrative metadata: who, when, project, facility, instrument.
- Technical metadata: instrument settings, method, file formats. Some of this is captured automatically.
- Descriptive metadata: sample, conditions, free-text notes. Users fill these in through forms generated from metadata templates (schemas) defined per instrument or collection.
This is what makes data findable months later: search by sample, operator, instrument or any template field.
5. Store and protect
Data land in institutional or national storage, such as Onedata spaces or S3 buckets, with access control based on your institutional identity (e-INFRA CZ AAI). Storage choice determines capacity, redundancy and backup guarantees. See the service terms of the respective storage.
6. Use and share
Collaborators access data where they compute. They can mount them on a workstation or HPC cluster, open them in the browser, or download them via links. Sharing is managed per project or dataset instead of by copying files around.
7. Publish or archive
When the data support a publication, a curated dataset is deposited in a repository such as the National Repository Platform (NRP) or Zenodo, and gets a persistent identifier (DOI). Data that are not published are kept or deleted according to the retention policy.
Identity and access
All three solutions can use e-INFRA CZ or institutional federated login (eduID.cz). Instruments and services authenticate with tokens that are restricted in scope (for example to one dataset folder) and in time. Users never type their personal passwords on shared instrument PCs.
