7-to-NeST: Survey File Management

File Management

I never had anyone specifically mention their file explorer as part of their software stack, but with the number of files some workflows generate, I think it deservers an honorary mention. There are multiple contributing factors to the volume of files:

• Different software packages often require files in a specific format
• Many processing packages act more as a collection of individual executables, where each one reads in files from a previous step and creates another set of files for the next step.

Most, if not all, of these files need to be organized and archived if the processing steps are to be recreatable in the future. This is especially important for when a portion or all of a seismic line needs to be reprocessed or tweaked as additional data and/or ground truth information is collected at a later date. Without a sound approach to file management, recreating/reprocessing existing results can quickly turn into a headache. I also suspect that this is one of the reasons that the processing of a project is most often a solo venture, as adding another person into the mix only makes organization more complex.

File creation and management throughout the workflow is so ingrained into the traditional seismic workflow, that most early testers of NeST initially struggled in various ways with our aim to handle all the files internally.

File Conversion & Manipulation

Every separate piece of software in the typical seismic processing workflow have at minimum files needed as inputs and files representing the result for that process in the workflow. That is how software works, why is this such an issue?

What happens when the resulting file coming from one software package can’t be imported into another? We have found that just because a file standard exists, like the various SEG file standards for waveform data, does not mean that the file standard is followed consistently by all who use it. This inevitably leads to issues when moving from one software to another, often neccesitating yet another tool or script to manipulate the file to include the header information or sidecar file needed for the next software package.

Adding additional information to files needed in downstream software is the most error prone step we have identified in the traditional workflow. The majority of all the SEG data we have used from a wide range of projects/sources from many different groups have had file header errors. Things like incorrect source and/or receiver locations, incorrect geo-referencing, and wrong timing information have been seen on multiple occasions. Some of these errors, especially uncaught geometry errors, can lead to serious issues in downstream processes that could render final results completely wrong.

All of the errors we found are likely from what I would consider minor lapses during fairly mundane steps in the workflow, likely when a script and/or spreadsheet was not updated from a previous line or project. These sorts of minor mistakes can be hard to identify within a complex disconnected workflow, especially when it lacks immediate validation and visulazation tools.

Survey File

At the base of the NeST data management infrastructure is what we call a Survey file. This survey file represents a single project/survey area and is a collection of individual Lines. Each Line represents a linear acquistion array that is ultimately processed to create 2D velocity (both Vp and Vs) estimates and various other derived results.

Saved within the Survey file is everything, and we really mean EVERYTHING, that is done to go from the raw field collected waveforms all the way through the final results. Some examples of the types of things saved internal to the Survey file include:

• QC steps including stacking, trace muting, channel mapping, geometry, GPS data, geo-referencing, and more
• Data analysis steps including processing flows, processing parameters, dispersion-curve picks, first-break picks, and more
• Inversion results
• Results including derived results and their associated parameters

Visualization tools

Humans have an inate ability to pick out patterns and discrepencies in images and plots, even more so when adding in the neccesary context. We try to leverage these abilities as much as possible by giving various linked data visualization tools for various steps. These tools allow for quick verification and validation of the data throughout the workflow. For instance, within our QC window we link our data selector across the following:

• Stack/shot selector
• Geometry tables
• 1-D Geometry profile graph
• Map based (Geo-referenced) profile
• Shot window

These linked visualization tools played a major role in identifying discrepencies and discovering most of the errors found in various files that I mentioned earlier. Powerful linked visualization tools are found throughout NeST, helping ensure data accuracy throughout the entire workflow.