A container for continuous data. More...

Data Structures
struct	MS3TraceSeg
	Container for a continuous trace segment, linkable. More...

struct	MS3TraceID
	Container for a trace ID, linkable. More...

struct	MS3TraceList
	Container for a collection of continuous trace segment, linkable. More...

struct	MS3Tolerance
	Callback functions that return time and sample rate tolerances. More...

Macros
#define	MSTRACEID_SKIPLIST_HEIGHT 8
	Maximum skip list height for MSTraceIDs.

#define	MS3Tolerance_INITIALIZER
	Initialializer for the tolerances MS3Tolerance.

#define	mstl3_addmsr(mstl, msr, splitversion, autoheal, flags, tolerance) mstl3_addmsr_recordptr (mstl, msr, NULL, splitversion, autoheal, flags, tolerance)
	Add a MS3Record to a MS3TraceList.

Functions
MS3TraceList *	mstl3_init (MS3TraceList *mstl)
	Initialize a MS3TraceList container.

void	mstl3_free (MS3TraceList **ppmstl, int8_t freeprvtptr)
	Free all memory associated with a MS3TraceList.

MS3TraceID *	mstl3_findID (MS3TraceList mstl, const char sid, uint8_t pubversion, MS3TraceID **prev)
	Find matching MS3TraceID in a MS3TraceList.

MS3TraceSeg *	mstl3_addmsr_recordptr (MS3TraceList mstl, const MS3Record msr, MS3RecordPtr *pprecptr, int8_t splitversion, int8_t autoheal, uint32_t flags, const MS3Tolerance tolerance)
	Add data coverage from an MS3Record to a MS3TraceList.

int64_t	mstl3_readbuffer (MS3TraceList *ppmstl, const char buffer, uint64_t bufferlength, int8_t splitversion, uint32_t flags, const MS3Tolerance *tolerance, int8_t verbose)
	Parse miniSEED from a buffer and populate a MS3TraceList.

int64_t	mstl3_readbuffer_selection (MS3TraceList *ppmstl, const char buffer, uint64_t bufferlength, int8_t splitversion, uint32_t flags, const MS3Tolerance tolerance, const MS3Selections selections, int8_t verbose)
	Parse miniSEED from a buffer and populate a MS3TraceList.

int64_t	mstl3_unpack_recordlist (MS3TraceID id, MS3TraceSeg seg, void *output, uint64_t outputsize, int8_t verbose)
	Unpack data samples in a Record List associated with a MS3TraceList.

int	mstl3_convertsamples (MS3TraceSeg *seg, char type, int8_t truncate)
	Convert the data samples associated with an MS3TraceSeg to another data type.

int	mstl3_resize_buffers (MS3TraceList *mstl)
	Resize data sample buffers of MS3TraceList to what is needed.

int64_t	mstl3_pack (MS3TraceList mstl, void(record_handler)(char , int, void ), void handlerdata, int reclen, int8_t encoding, int64_t packedsamples, uint32_t flags, int8_t verbose, char *extra)
	Pack MS3TraceList data into miniSEED records.

void	mstl3_printtracelist (const MS3TraceList *mstl, ms_timeformat_t timeformat, int8_t details, int8_t gaps, int8_t versions)
	Print trace list summary information for a MS3TraceList.

void	mstl3_printsynclist (const MS3TraceList mstl, const char dccid, ms_subseconds_t subseconds)
	Print SYNC trace list summary information for a MS3TraceList.

void	mstl3_printgaplist (const MS3TraceList mstl, ms_timeformat_t timeformat, double mingap, double *maxgap)
	Print gap/overlap list summary information for a MS3TraceList.

Detailed Description

A container for continuous data.

Trace lists are a container to organize continuous segments of data. By combining miniSEED data records into trace lists, the time series is reconstructed and ready for processing or summarization.

A trace list container starts with an MS3TraceList, which contains one or more MS3TraceID entries, which each contain one or more MS3TraceSeg entries. The MS3TraceID and MS3TraceSeg entries are easily traversed as linked structures.

The overall structure is illustrated as:

MS3TraceList
- MS3TraceID
  - MS3TraceSeg
  - MS3TraceSeg
  - ...
- MS3TraceID
  - MS3TraceSeg
  - MS3TraceSeg
  - ...
- ...

See also: ms3_readtracelist(); ms3_readtracelist_timewin(); ms3_readtracelist_selection(); mstl3_writemseed()

Data Structure Documentation

◆ MS3TraceSeg

struct MS3TraceSeg

Container for a continuous trace segment, linkable.

Data Fields
nstime_t	starttime	Time of first sample.
nstime_t	endtime	Time of last sample.
double	samprate	Nominal sample rate (Hz)
int64_t	samplecnt	Number of samples in trace coverage.
void *	datasamples	Data samples, numsamples of type sampletype.
uint64_t	datasize	Size of datasamples buffer in bytes.
int64_t	numsamples	Number of data samples in datasamples.
char	sampletype	Sample type code, see Sample Types.
void *	prvtptr	Private pointer for general use, unused by library.
struct MS3RecordList *	recordlist	List of pointers to records that contributed.
struct MS3TraceSeg *	prev	Pointer to previous segment.
struct MS3TraceSeg *	next	Pointer to next segment, NULL if the last.

◆ MS3TraceID

struct MS3TraceID

Container for a trace ID, linkable.

Data Fields
char	sid[LM_SIDLEN]	Source identifier as URN, max length LM_SIDLEN.
uint8_t	pubversion	Largest contributing publication version.
nstime_t	earliest	Time of earliest sample.
nstime_t	latest	Time of latest sample.
void *	prvtptr	Private pointer for general use, unused by library.
uint32_t	numsegments	Number of segments for this ID.
struct MS3TraceSeg *	first	Pointer to first of list of segments.
struct MS3TraceSeg *	last	Pointer to last of list of segments.
struct MS3TraceID *	next[MSTRACEID_SKIPLIST_HEIGHT]	Next trace ID at first pointer, NULL if the last.
uint8_t	height	Height of skip list at next.

◆ MS3TraceList

struct MS3TraceList

Container for a collection of continuous trace segment, linkable.

Data Fields
uint32_t	numtraceids	Number of traces IDs in list.
struct MS3TraceID	traces	Head node of trace skip list, first entry at traces.next[0].
uint64_t	prngstate	INTERNAL: State for Pseudo RNG.

◆ MS3Tolerance

struct MS3Tolerance

Callback functions that return time and sample rate tolerances.

A container for function pointers that return time and sample rate tolerances that are used for merging data into MS3TraceList containers. The functions are provided with a MS3Record and must return the acceptable tolerances to merge this with other data.

The time(MS3Record) function must return a time tolerance in seconds.

The samprate(MS3Record) function must return a sampling rate tolerance in Hertz.

For any function pointer set to NULL a default tolerance will be used.

Illustrated usage:

MS3Tolerance tolerance;
 
tolerance.time = my_time_tolerance_function;
tolerance.samprate = my_samprate_tolerance_function;
 
mstl3_addmsr (mstl, msr, 0, 1, &tolerance);

See also: mstl3_addmsr()

Data Fields
double()(const MS3Record msr)	time	Pointer to function that returns time tolerance.
double()(const MS3Record msr)	samprate	Pointer to function that returns sample rate tolerance.

Macro Definition Documentation

◆ MS3Tolerance_INITIALIZER

#define MS3Tolerance_INITIALIZER

Value:

  {                                \
    .time = NULL, .samprate = NULL \
  }

Initialializer for the tolerances MS3Tolerance.

◆ mstl3_addmsr

#define mstl3_addmsr	(	mstl,
		msr,
		splitversion,
		autoheal,
		flags,
		tolerance ) mstl3_addmsr_recordptr (mstl, msr, NULL, splitversion, autoheal, flags, tolerance)

Add a MS3Record to a MS3TraceList.

See also: mstl3_addmsr_recordptr()

Function Documentation

◆ mstl3_init()

MS3TraceList * mstl3_init ( MS3TraceList * mstl )

extern

Initialize a MS3TraceList container.

A new MS3TraceList is allocated if needed. If the supplied MS3TraceList is not NULL it will be re-initialized and any associated memory will be freed including data at prvtptr pointers.

Parameters

[in] mstl MS3TraceList to reinitialize or NULL

Returns: a pointer to a MS3TraceList struct on success or NULL on error.

MessageOnError - this function logs a message on error

◆ mstl3_free()

void mstl3_free	(	MS3TraceList **	ppmstl,
		int8_t	freeprvtptr )

extern

Free all memory associated with a MS3TraceList.

The pointer to the target MS3TraceList will be set to NULL.

Parameters

[in]	ppmstl	Pointer-to-pointer to the target MS3TraceList to free
[in]	freeprvtptr	If true, also free any data at the prvtptr members of MS3TraceID.prvtptr, MS3TraceSeg.prvtptr, and MS3RecordPtr.prvtptr (in MS3TraceSeg.recordlist)

◆ mstl3_findID()

MS3TraceID * mstl3_findID	(	MS3TraceList *	mstl,
		const char *	sid,
		uint8_t	pubversion,
		MS3TraceID **	prev )

extern

Find matching MS3TraceID in a MS3TraceList.

Return the MS3TraceID matching the sid in the specified MS3TraceList.

If prev is not NULL, set pointers to previous entries for the expected location of the trace ID. Useful for adding a new ID with mstl3_addID(), and should be set to NULL otherwise.

Parameters

[in]	mstl	Pointer to the MS3TraceList to search
[in]	sid	Source ID to search for in the list
[in]	pubversion	If non-zero, find the entry with this version
[out]	prev	Pointers to previous entries in expected location or NULL

Returns: a pointer to the matching MS3TraceID or NULL if not found or error.

◆ mstl3_addmsr_recordptr()

MS3TraceSeg * mstl3_addmsr_recordptr	(	MS3TraceList *	mstl,
		const MS3Record *	msr,
		MS3RecordPtr **	pprecptr,
		int8_t	splitversion,
		int8_t	autoheal,
		uint32_t	flags,
		const MS3Tolerance *	tolerance )

extern

Add data coverage from an MS3Record to a MS3TraceList.

Searching the list for the appropriate MS3TraceID and MS3TraceSeg and either add data to existing entries or create new ones as needed.

As this routine adds data to a trace list it constructs continuous time series, merging segments when possible. The tolerance pointer to a MS3Tolerance identifies function pointers that are expected to return time tolerace, in seconds, and sample rate tolerance, in Hertz, for the given MS3Record. If tolerance is NULL, or the function pointers it identifies are NULL, default tolerances will be used as follows: - Default time tolerance is 1/2 the sampling period - Default sample rate (sr) tolerance is abs(1‐sr1/sr2) < 0.0001

In recommended usage, the splitversion flag is 0 and different publication versions of otherwise matching data are merged. If more than one version contributes to a given source identifer's segments, its MS3TraceID.pubversion will be the set to the largest contributing version. If the splitversion flag is 1 the publication versions will be kept separate with each version isolated in separate MS3TraceID entries.

If the autoheal flag is true, extra processing is invoked to conjoin trace segments that fit together after the MS3Record coverage is added. For segments that are removed, any memory at the MS3TraceSeg.prvtptr will be freed.

If the pprecptr is not NULL a Record List will be maintained for each segment. If the value of *pprecptr is NULL, a new MS3RecordPtr will be allocated, otherwise the supplied structure will be updated. The MS3RecordPtr will be added to the appropriate record list and the values of MS3RecordPtr.msr and MS3RecordPtr.endtime will be set, all other fields should be set by the caller.

The lists are always maintained in a sorted order. An MS3TraceList is maintained with the MS3TraceID entries in ascending alphanumeric order of SID. If repeated SIDs are present due to splitting on publication version, they are maintained in order of ascending version. A MS3TraceID is always maintained with MS3TraceSeg entries in data time time order.

Parameters

[in]	mstl	Destination MS3TraceList to add data to
[in]	msr	MS3Record containing the data to add to list
[in]	pprecptr	Pointer to pointer to a MS3RecordPtr for Record List
[in]	splitversion	Flag to control splitting of version/quality
[in]	autoheal	Flag to control automatic merging of segments
[in]	flags	Flags to control optional functionality (unused)
[in]	tolerance	Tolerance function pointers as MS3Tolerance

Returns: a pointer to the MS3TraceSeg updated or NULL on error.

See also: mstl3_readbuffer(); ms3_readtracelist()

MessageOnError - this function logs a message on error

◆ mstl3_readbuffer()

int64_t mstl3_readbuffer	(	MS3TraceList **	ppmstl,
		const char *	buffer,
		uint64_t	bufferlength,
		int8_t	splitversion,
		uint32_t	flags,
		const MS3Tolerance *	tolerance,
		int8_t	verbose )

extern

Parse miniSEED from a buffer and populate a MS3TraceList.

For a full description of tolerance see mstl3_addmsr().

If the MSF_UNPACKDATA flag is set in flags, the data samples will be unpacked. In most cases the caller probably wants this flag set, without it the trace list will merely be a list of channels.

If the MSF_RECORDLIST flag is set in flags, a MS3RecordList will be built for each MS3TraceSeg. The MS3RecordPtr entries contain the location of the data record, bit flags, extra headers, etc.

Parameters

[in]	ppmstl	Pointer-to-point to destination MS3TraceList
[in]	buffer	Source buffer to read miniSEED records from
[in]	bufferlength	Maximum length of buffer
[in]	splitversion	Flag to control splitting of version/quality
[in]	flags	Flags to control parsing and optional functionality: `MSF_RECORDLIST` : Build a MS3RecordList for each MS3TraceSeg Flags supported by msr3_parse() Flags supported by mstl3_addmsr()
[in]	tolerance	Tolerance function pointers as MS3Tolerance
[in]	verbose	Controls verbosity, 0 means no diagnostic output

Returns: The number of records parsed on success, otherwise a negative library error code.

MessageOnError - this function logs a message on error

See also: mstl3_addmsr()

◆ mstl3_readbuffer_selection()

int64_t mstl3_readbuffer_selection	(	MS3TraceList **	ppmstl,
		const char *	buffer,
		uint64_t	bufferlength,
		int8_t	splitversion,
		uint32_t	flags,
		const MS3Tolerance *	tolerance,
		const MS3Selections *	selections,
		int8_t	verbose )

extern

Parse miniSEED from a buffer and populate a MS3TraceList.

For a full description of tolerance see mstl3_addmsr().

If the MSF_UNPACKDATA flag is set in flags, the data samples will be unpacked. In most cases the caller probably wants this flag set, without it the trace list will merely be a list of channels.

If the MSF_RECORDLIST flag is set in flags, a MS3RecordList will be built for each MS3TraceSeg. The MS3RecordPtr entries contain the location of the data record, bit flags, extra headers, etc.

If selections is not NULL, the MS3Selections will be used to limit what is returned to the caller. Any data not matching the selections will be skipped.

Parameters

[in]	ppmstl	Pointer-to-point to destination MS3TraceList
[in]	buffer	Source buffer to read miniSEED records from
[in]	bufferlength	Maximum length of buffer
[in]	splitversion	Flag to control splitting of version/quality
[in]	flags	Flags to control parsing and optional functionality: `MSF_RECORDLIST` : Build a MS3RecordList for each MS3TraceSeg Flags supported by msr3_parse() Flags supported by mstl3_addmsr()
[in]	tolerance	Tolerance function pointers as MS3Tolerance
[in]	selections	Specify limits to which data should be returned, see Data Selections
[in]	verbose	Controls verbosity, 0 means no diagnostic output

Returns: The number of records parsed on success, otherwise a negative library error code.

MessageOnError - this function logs a message on error

See also: mstl3_addmsr()

◆ mstl3_unpack_recordlist()

int64_t mstl3_unpack_recordlist	(	MS3TraceID *	id,
		MS3TraceSeg *	seg,
		void *	output,
		uint64_t	outputsize,
		int8_t	verbose )

extern

Unpack data samples in a Record List associated with a MS3TraceList.

Normally a record list is created calling by ms3_readtracelist() or mstl3_readbuffer() with the MSF_RECORDLIST flag.

Unpacked data samples are written to the provided output buffer (up to outputsize bytes). If output is NULL, a buffer will be allocated and associated with the MS3TraceSeg, just as if the data were unpacked while constructing the Trace List.

The sample type of the decoded data is stored at MS3TraceSeg.sampletype (i.e. seg->sampletype).

A record pointer entry has multiple ways to identify the location of a record: memory buffer, open file (FILE *) or file name. This routine uses the first populated record location in the following order:

Buffer pointer (MS3RecordPtr.bufferptr)
Open file and offset (MS3RecordPtr.fileptr and MS3RecordPtr.fileoffset)
File name and offset (MS3RecordPtr.filename and MS3RecordPtr.fileoffset)

It would be unusual to build a record list outside of the library, but should that ever occur note that the record list is assumed to be in correct time order and represent a contiguous time series.

Parameters

[in]	id	MS3TraceID for relevant MS3TraceSeg
[in]	seg	MS3TraceSeg with associated Record List to unpack
[out]	output	Output buffer for data samples, can be NULL
[in]	outputsize	Size of output buffer
[in]	verbose	Controls logging verbosity, 0 is no diagnostic output

Returns: the number of samples unpacked or -1 on error.

MessageOnError - this function logs a message on error

See also: mstl3_readbuffer(); mstl3_readbuffer_selection(); ms3_readtracelist(); ms3_readtracelist_selection()

◆ mstl3_convertsamples()

int mstl3_convertsamples	(	MS3TraceSeg *	seg,
		char	type,
		int8_t	truncate )

extern

Convert the data samples associated with an MS3TraceSeg to another data type.

Text data samples cannot be converted, if supplied or requested an error will be returned.

When converting float & double sample types to integer type a simple rounding is applied by adding 0.5 to the sample value before converting (truncating) to integer. This compensates for common machine representations of floating point values, e.g. "40.0" represented by "39.99999999".

If the truncate flag is true (non-zero) data samples will be truncated to integers even if loss of sample precision is detected. If the truncate flag is false (zero) and loss of precision is detected an error is returned. Loss of precision is determined by testing that the difference between the floating point value and the (truncated) integer value is greater than 0.000001.

Parameters

[in]	seg	The target MS3TraceSeg to convert
[in]	type	The desired data sample type: `'i'` - 32-bit integer data type `'f'` - 32-bit float data type `'d'` - 64-bit float (double) data type
[in]	truncate	Control truncation of floating point values to integers

Returns: 0 on success, and -1 on failure.

MessageOnError - this function logs a message on error

◆ mstl3_resize_buffers()

int mstl3_resize_buffers ( MS3TraceList * mstl )

extern

Resize data sample buffers of MS3TraceList to what is needed.

This routine should only be used if pre-allocation of memory, via libmseed_prealloc_block_size, was enabled to allocate the buffers.

Parameters

[in] mstl MS3TraceList to resize buffers

Returns: Return 0 on success, otherwise returns a libmseed error code.

MessageOnError - this function logs a message on error

◆ mstl3_pack()

int64_t mstl3_pack	(	MS3TraceList *	mstl,
		void(*	record_handler )(char , int, void ),
		void *	handlerdata,
		int	reclen,
		int8_t	encoding,
		int64_t *	packedsamples,
		uint32_t	flags,
		int8_t	verbose,
		char *	extra )

extern

Pack MS3TraceList data into miniSEED records.

The datasamples array, numsamples and starttime fields of each trace segment will be adjusted as data are packed unless the MSF_MAINTAINMSTL flag is specified in flags. If MSF_MAINTAINMSTL is specified a caller would also normally set the MSF_FLUSHDATA flag to pack all data in the trace list.

Use as a rolling buffer to generate data records: The behavior of adjusting the trace list as data are packed is intended to allow using a MS3TraceList as an intermediate collection of data buffers to generate data records from an arbitrarily large data source, e.g. continuous data. In this pattern, data are added to a MS3TraceList and mstl3_pack() is called repeatedly. Data records are only produced if a complete record can be generated, which often leaves small amounts of data in each segment buffer. On completion or shutdown the caller usually makes a final call to mst3_pack() with the MSF_FLUSHDATA flag set to flush all data from the buffers.

As each record is filled and finished they are passed to record_handler() which should expect 1) a char* to the record, 2) the length of the record and 3) a pointer supplied by the original caller containing optional private data (handlerdata). It is the responsibility of record_handler() to process the record, the memory will be re-used or freed when record_handler() returns.

The requested encoding value is currently only used for integer data samples. The encoding is set automatially for text and floating point data samples as there is only a single encoding for them. A value of -1 can be used to request the default.

If extra is not NULL it is expected to contain extraheaders, a string containing (compact) JSON, that will be added to each output record.

Parameters

[in]	mstl	MS3TraceList containing data to pack
[in]	record_handler()	Callback function called for each record
[in]	handlerdata	A pointer that will be provided to the record_handler()
[in]	reclen	Maximum record length to create
[in]	encoding	Encoding for data samples, see msr3_pack()
[out]	packedsamples	The number of samples packed, returned to caller
[in]	flags	Bit flags to control packing: `MSF_FLUSHDATA` : Pack all data in the buffer `MSF_MAINTAINMSTL` : Do not remove packe data from the buffer `MSF_PACKVER2` : Pack miniSEED version 2 instead of default 3
[in]	verbose	Controls logging verbosity, 0 is no diagnostic output
[in]	extra	If not NULL, add this buffer of extra headers to all records

Returns: the number of records created on success and -1 on error.

MessageOnError - this function logs a message on error

See also: msr3_pack()

◆ mstl3_printtracelist()

void mstl3_printtracelist	(	const MS3TraceList *	mstl,
		ms_timeformat_t	timeformat,
		int8_t	details,
		int8_t	gaps,
		int8_t	versions )

extern

Print trace list summary information for a MS3TraceList.

By default only print the source ID, starttime and endtime for each trace. If details is greater than 0 include the sample rate, number of samples and a total trace count. If gaps is greater than 0 and the previous trace matches (SID & samprate) include the gap between the endtime of the last trace and the starttime of the current trace.

Parameters

[in]	mstl	MS3TraceList to print
[in]	timeformat	Time string format, one of ms_timeformat_t
[in]	details	Flag to control inclusion of more details
[in]	gaps	Flag to control inclusion of gap/overlap between segments
[in]	versions	Flag to control inclusion of publication version on SourceIDs

◆ mstl3_printsynclist()

void mstl3_printsynclist	(	const MS3TraceList *	mstl,
		const char *	dccid,
		ms_subseconds_t	subseconds )

extern

Print SYNC trace list summary information for a MS3TraceList.

The SYNC header line will be created using the supplied dccid, if the pointer is NULL the string "DCC" will be used instead.

If the subsecond flag is true the segment start and end times will include subsecond precision, otherwise they will be truncated to integer seconds.

Parameters

[in]	mstl	MS3TraceList to print
[in]	dccid	The DCC identifier to include in the output
[in]	subseconds	Inclusion of subseconds, one of ms_subseconds_t

◆ mstl3_printgaplist()

void mstl3_printgaplist	(	const MS3TraceList *	mstl,
		ms_timeformat_t	timeformat,
		double *	mingap,
		double *	maxgap )

extern

Print gap/overlap list summary information for a MS3TraceList.

Overlaps are printed as negative gaps.

If mingap and maxgap are not NULL their values will be enforced and only gaps/overlaps matching their implied criteria will be printed.

Parameters

[in]	mstl	MS3TraceList to print
[in]	timeformat	Time string format, one of ms_timeformat_t
[in]	mingap	Minimum gap to print in seconds (pointer to value)
[in]	maxgap	Maximum gap to print in seconds (pointer to value)

Data Structures

Macros

Functions

Detailed Description

Data Structure Documentation

◆ MS3TraceSeg

◆ MS3TraceID

◆ MS3TraceList

◆ MS3Tolerance

Macro Definition Documentation

◆ MS3Tolerance_INITIALIZER

◆ mstl3_addmsr

Function Documentation

◆ mstl3_init()

◆ mstl3_free()

◆ mstl3_findID()

◆ mstl3_addmsr_recordptr()

◆ mstl3_readbuffer()

◆ mstl3_readbuffer_selection()

◆ mstl3_unpack_recordlist()

◆ mstl3_convertsamples()

◆ mstl3_resize_buffers()

◆ mstl3_pack()

◆ mstl3_printtracelist()

◆ mstl3_printsynclist()

◆ mstl3_printgaplist()