tmtk - TranSMART data curation toolkit

Author:Jochem Bijlard Source Code:https://github.com/thehyve/tmtk/ Generated:Jan 29, 2018 License:GPLv3 Version:0.2.0

Philosophy

A toolkit for ETL curation for the tranSMART data warehouse for translational research.

The TranSMART curation toolkit (tmtk) aims to provide a language and set of classes for describing data to be uploaded to tranSMART. The toolkit can be used to edit and validate studies prior to loading them with transmart-batch.

Functionality currently available:

• create a transmart-batch ready study from clinical data files.
• load an existing study and validate its contents.
• edit the transmart concept tree in The Arborist graphical editor.
• create chromosomal region annotation files.
• map HGNC gene symbols to corresponding Entrez gene IDs using mygene.info.

Note

tmtk is a python3 package meant to be run in Jupyter notebooks. Results for other setups may vary.

Contents

Changelog

Version 0.2.0

• Create and apply tree templates in Arborist
• Improved interaction with metadata tags in Arborist
• Resolved issues with the validator
• R is now an optional dependency

User examples

These examples have been extracted from Jupyter Notebooks.

Create study from clinical data.

tmtk has a wizard that can be used to quickly go from clinical data files to a study object. The main goal of this functionality is to reduce the barrier of setting up all transmart-batch specific files (i.e. parameter files, column mapping and word mapping files).

The way to use it is to call tmtk.wizard.create_study(path), where path points a directory with clinical data files.

Note: clinical datafiles have to be in a format that is accepted by transmart-batch.

Here we will create a study from these two files:

import os
files_dir = './studies/wizard/'
os.listdir(files_dir)

['Cell-line_clinical.txt', 'Cell-line_NHTMP.txt']

# Load the toolkit
import tmtk

# Create a study object by running the wizard
study = tmtk.wizard.create_study('./studies/wizard/')

==========  study params  ==========
Updating:  TOP_NODE=(Public|Private) Studies<STUDY_ID> (default).
Helptext:  The study top node.
Change to:  Public StudiesYou're a wizard Harry
Updating! (TOP_NODE=Public StudiesYou're a wizard Harry)
--------------------
Updating:  STUDY_ID=Uppercased parent directory.
Helptext:  Identifier of the study.
Change to:  WIZARD
Updating! (STUDY_ID=WIZARD)
--------------------
Updating:  SECURITY_REQUIRED=N (default).
Helptext:  Defines study as Private (Y) or Public (N).
Change to:  N
Updating! (SECURITY_REQUIRED=N)
--------------------
==========  clinical params  ==========
Updating:  TAGS_FILE (not set)
Helptext:  Points to the concepts tags file.
Change to:
--------------------
Updating:  WORD_MAP_FILE (not set)
Helptext:  Points to the file with dictionary to be used.
Change to:
--------------------
Updating:  COLUMN_MAP_FILE (not set)
Helptext:  Points to the column file.
Change to:
--------------------
Updating:  XTRIAL_FILE (not set)
Helptext:  Points to the cross study concepts file.
Change to:
--------------------

#####  Please select your clinical datafiles  #####
-    0. /Users/jb/thehyve/tmtk/studies/wizard/Cell-line_clinical.txt
-    1. /Users/jb/thehyve/tmtk/studies/wizard/Cell-line_NHTMP.txt
Pick number:  0
Selected files: ['Cell-line_clinical.txt']
Pick number:  1
Selected files: ['Cell-line_clinical.txt', 'Cell-line_NHTMP.txt']
Pick number:

✅ Adding 'Cell-line_clinical.txt' as clinical datafile to study.

✅ Adding 'Cell-line_NHTMP.txt' as clinical datafile to study.

---

The wizard walked us through some of the options for the study we want to create. Our new study is a public study with STUDY_ID==WIZARD and an appropriate name. None of the clinical params have been set, so tmtk will use default names for the column and word mapping file. Next the datafiles have been loaded and the column mapping object has been created to include the data files.

Next we will run the validator and find out that some files cannot be found. This is expected as these objects are only in memory and not yet on disk.

study.validate_all(5)

⚠ No valid file found on disk for /Users/jb/thehyve/tmtk/studies/wizard/clinical/word_mapping_file.txt, creating dataframe.

Validating params file at clinical

❌ WORD_MAP_FILE=word_mapping_file.txt cannot be found.

❌ COLUMN_MAP_FILE=column_mapping_file.txt cannot be found.

Detected parameter WORD_MAP_FILE=word_mapping_file.txt.

Detected parameter COLUMN_MAP_FILE=column_mapping_file.txt.

Validating params file at study

Detected parameter TOP_NODE=\Public Studies\You're a wizard Harry\.

Detected parameter STUDY_ID=WIZARD.

Detected parameter SECURITY_REQUIRED=N.

---

Of course, we want to write our study to disk so it can be loaded with transmart-batch.

study.write_to('~/studies/my_new_study')

Writing file to /Users/jb/studies/my_new_study/clinical/clinical.params

Writing file to /Users/jb/studies/my_new_study/study.params

Writing file to /Users/jb/studies/my_new_study/clinical/column_mapping_file.txt

Writing file to /Users/jb/studies/my_new_study/clinical/Cell-line_clinical.txt

Writing file to /Users/jb/studies/my_new_study/clinical/word_mapping_file.txt

Writing file to /Users/jb/studies/my_new_study/clinical/Cell-line_NHTMP.txt

---

TranSMART Arborist

GUI editor for the concept tree.

First load the toolkit.

import tmtk

Create a study object by entering a “study.params” file.

study = tmtk.Study('../studies/valid_study/study.params')

To verify the study object is compatible with transmart-batch for loading you can run the validator

study.validate_all()

Validating Tags:

❌ Tags (2) found that cannot map to tree: (1. Cell line characteristics∕1. Cell lines∕Age and 1. Cell line characteristics∕1. Cell lines∕Gender). You might want to call_boris() to fix them.

We will ignore this issue for now as this will be fixed automatically when calling the Arborist GUI.

The GUI allows a user to interactively edit all aspects of TranSMART’s concept tree, this include:

• Concept Paths from the clinical column mapping.
• Word mapping from clinical data files.
• High dimensional paths from subject sample mapping files.
• Meta data tags

# In a Jupyter Notebook, this brings up the interactive concept tree editor.
study.call_boris()

Once returned from The Arborist to Jupyter environment we can write the updated files to disk. You can then run transmart-batch on that study to load it into your tranSMART instance.

study.save_to('~/studies/updated_study')

Collaboration with non technical users.

Though using Jupyter Notebooks is great for technical users, less technical domain experts might quickly feel discouraged. To allow for collaboration with these users we will upload this concept tree to a running Boris as a Service webserver. This will allow others to make refinements to the concept tree.

study.publish_to_baas('arborist-test-trait.thehyve.net')

Once the study is updated in BaaS, we can update the local files by copying the url for the latest tree into this command.

study.update_from_baas('arborist-test-trait.thehyve.net/trees/valid-study/3/~edit')

API Description

Study class

class tmtk.Study(study_params_path=None, minimal=False)

Bases: object

Describes an entire TranSMART study. This is the main object used in tmtk. Studies can be initialized by pointing to a study.params file. This study has to be structured according to specification for transmart-batch.

>>> import tmtk
>>> study = tmtk.Study('./studies/valid_study/study.params')

This will create the study object which can be used as a starting point for custom curation or directly in The Arborist.

add_clinical(clinical_params)

Add clinical data to a study object.

Parameters:clinical_params – tmkt.ClinicalParams object

add_metadata()

Create the Tags object for this study. Does nothing if it is already present.

call_boris(height=650)

Launch The Arborist GUI editor for the concept tree. This starts a Flask webserver in an IFrame when running in a Jupyter Notebook.

While The Arborist is opened, the GIL prevents any other actions. :param height: set the height of the output cell

concept_tree

ConceptTree object for this study.

concept_tree_json

Stringified JSON that is used by JSTree in The Arborist.

concept_tree_to_clipboard()

Send stringified JSON that is used by JSTree in The Arborist to clipboard.

files_with_changes()

Find dataframes that have changed since they have been loaded.

find_annotation(platform=None)

Search for annotation data with this study and return it.

Parameters:platform – platform id to look for in this study. Returns:an Annotations object or nothing.

find_params_for_datatype(datatypes=None)

Search for parameter files within this study object and return them as list.

Parameters:datatypes – single string datatype or list of strings Returns:a list of parameter objects for specific datatype in this study

get_objects_with_prop(prop: <built-in function all>)

Search for objects with a certain property.

Parameters:prop – string equal to the property name. Returns:generator for the found objects.

high_dim_nodes

All subject sample mapping object in this study.

publish_to_baas(url, study_name=None, username=None)

Publishes a tree on a Boris as a Service instance.

Parameters:

• url – url to a instance (e.g. http://transmart-arborist.thehyve.nl/).
• study_name – a nice name.
• username – if no username is given, you will be prompted for one.

Returns:

the url that points to the study you’ve just uploaded.

study_id

The study ID as it is set in study params.

study_name

The study name, extracted from study param TOP_NODE.

update_from_baas(url, username=None)

Give url to a tree in BaaS.

Parameters:

• url – url that has both the study and version of a tree in BaaS (e.g. http://transmart-arborist.thehyve.nl/trees/study-name/1/~edit/).
• username – if no username is given, you will be prompted for one.

update_from_treefile(treefile)

Give path to a treefile (from Boris as a Service or otherwise) and update the current study to match made changes.

Parameters:treefile – path to a treefile (stringified JSON).

validate_all(verbosity: int = 2)

Validate all items in this study.

Parameters:verbosity – set the verbosity of output, pick 0, 1, 2, 3 or 4. Returns:True if everything is okay, else return False.

write_to(root_dir, overwrite=False)

Write this study to a new directory on file system.

Parameters:

• root_dir – the base directory to write the study to.
• overwrite – set this to True to overwrite existing files.

---

Params classes

Params Container

class tmtk.params.Params.Params(study_folder=None)

Bases: object

Container class for all params files, called by Study to locate all params files.

add_params(path, parameters=None)

Add a new parameter file to the Params object.

Parameters:

• path – a path to a parameter file.
• parameters – add dict here with parameters if you want to create a new parameter file.

static create_params(path, parameters=None, subdir=None)

Create a new parameter file object.

Parameters:

• path – a path to a parameter file.
• parameters – add dict here with parameters if you want to create a new parameter file.
• subdir – subdir is used as string representation.

Returns:

parameter file object.

validate_all(verbosity=3)

Base class: ParamsBase

class tmtk.params.ParamsBase.ParamsBase(path=None, parameters=None, subdir=None, parent=None)

Bases: object

Base class for parameter files.

get(parameter, default=None)

Return value for parameter.

Parameters:

• parameter – string will be converted to uppercase.
• default – return default if value is not found.

Returns:

value for this parameter if set, else None.

save()

Overwrite the original file with the current parameters.

update()

Iterate over parameters to change them interactively.

validate(verbosity=3)

Validate this parameter file. Return True if no errors were found in this parameter file.

Parameters:verbosity – set the verbosity of output, pick 0, 1, 2, 3 or 4. Returns:True or False if errors are found.

write_to(path, overwrite=False)

Writes parameters in object to file in path. Does not overwrite existing files unless specifically told.

Parameters:

• path – path to store parameters to.
• overwrite – allow overwriting existing files.

AnnotationParams

class tmtk.params.AnnotationParams.AnnotationParams(path=None, parameters=None, subdir=None, parent=None)

Bases: tmtk.params.ParamsBase.ParamsBase

is_viable()

Returns:True if both the platform is set and the annotations file is located, else returns False.

mandatory

optional

ClinicalParams

class tmtk.params.ClinicalParams.ClinicalParams(path=None, parameters=None, subdir=None, parent=None)

Bases: tmtk.params.ParamsBase.ParamsBase

docslink = 'https://github.com/thehyve/transmart-batch/blob/master/docs/clinical.md'

is_viable()

Returns:True if both the column mapping file is located, else returns False.

mandatory

optional

HighDimParams

class tmtk.params.HighDimParams.HighDimParams(path=None, parameters=None, subdir=None, parent=None)

Bases: tmtk.params.ParamsBase.ParamsBase

docslink = 'https://github.com/thehyve/transmart-batch/blob/master/docs/hd-params.md'

is_viable()

Returns:True if both the datafile and map file are located, else returns False.

mandatory

optional

StudyParams

class tmtk.params.StudyParams.StudyParams(path=None, parameters=None, subdir=None, parent=None)

Bases: tmtk.params.ParamsBase.ParamsBase

docslink = 'https://github.com/thehyve/transmart-batch/blob/master/docs/study-params.md'

is_viable()

Returns:True if STUDY_ID has been set.

mandatory

optional

TagsParams

class tmtk.params.TagsParams.TagsParams(path=None, parameters=None, subdir=None, parent=None)

Bases: tmtk.params.ParamsBase.ParamsBase

is_viable()

Returns:True if both the column mapping file is located, else returns False.

mandatory

optional

---

Clinical classes

Clinical Container

class tmtk.clinical.Clinical(clinical_params)

Bases: object

Container class for all clinical data related objects, i.e. the column mapping, word mapping, and clinical data files.

This object has methods that add data files, and for lookups of clinical files and variables.

add_datafile(filename, dataframe=None)

Add a clinical data file to study.

Parameters:

• filename – path to file or filename of file in clinical directory.
• dataframe – if given, add pd.DataFrame to study.

all_variables

Dictionary where {tmtk.VarID: tmtk.Variable} for all variables in the column mapping file.

call_boris(height=650)

Use The Arborist to modify only information in the column and word mapping files. :param height: set the height of the output cell

get_datafile(name: str)

Find datafile object by filename.

Parameters:name – name of file. Returns:tmtk.DataFile object.

get_variable(var_id: tuple)

Return a Variable object based on variable id.

Parameters:var_id – tuple of filename and column number. Returns:tmtk.Variable.

show_changes()

Print changes made to the column mapping and word mapping file.

validate_all(verbosity=3)

ColumnMapping

class tmtk.clinical.ColumnMapping(params=None)

Bases: tmtk.utils.filebase.FileBase

Class with utilities for the column mapping file for clinical data. Can be initiated with by giving a clinical params file object.

append_from_datafile(datafile)

Appends the column mapping file with rows based on datafile column names.

Parameters:datafile – tmtk.DataFile object.

build_index(df=None)

Build index for the column mapping dataframe. If pd.DataFrame (optional) is given, modify and return that.

Parameters:df – pd.DataFrame. Returns:pd.DataFrame.

call_boris(height=650)

Use The Arborist to modify only information in the column mapping file. :param height: set the height of the output cell

create_df()

Create pd.DataFrame with a correct header.

Returns:pd.DataFrame.

get_concept_path(var_id: tuple)

Return concept path for given variable identifier tuple.

Parameters:var_id – tuple of filename and column number. Return str:concept path for this variable.

ids

A list of variable identifier tuples.

included_datafiles

List of datafiles included in column mapping file.

path_changes(silent=False)

Determine changes made to column mapping file.

Parameters:silent – if True, only print output. Returns:if silent=False return dictionary with changes since load.

path_id_dict

Dictionary with all variable ids as keys and paths as value.

select_row(var_id: tuple)

Select row based on variable identifier tuple. Raises exception if variable is not in this column mapping.

Parameters:var_id – tuple of filename and column number. Returns:list of items in selected row.

validate(verbosity=2)

DataFile

class tmtk.clinical.DataFile(path=None)

Bases: tmtk.utils.filebase.FileBase

Class for clinical data files, does not do much more than tmkt.FileBase.

validate(verbosity=2)

Variable

class tmtk.clinical.Variable(datafile, column: int = None, clinical_parent=None)

Bases: object

Base class for clinical variables

column_map_data

Column mapping row as dictionary where keys are short descriptors.

Returns:dict.

concept_path

Concept path after conversions by transmart-batch.

Returns:str.

data_label

Variable data label.

Returns:str.

forced_categorical

Check if forced categorical by entering ‘CATEGORICAL’ in 7th column.

Returns:bool.

is_empty

Check if variable is fully empty.

Returns:bool.

is_in_wordmap

Check if variable is represented in word mapping file.

Returns:bool.

is_numeric

True if transmart-batch will load this concept as numerical. This includes information from word mapping and column mapping.

Returns:bool.

is_numeric_in_datafile

True if the datafile contains only numerical items.

Returns:bool.

mapped_values

Data items after word mapping.

Returns:list.

unique_values

Returns:Unique set of values in the datafile.

validate(verbosity=2)

values

Returns:All values as found in the datafile.

var_id

Returns:Variable identifier tuple (datafile.name, column).

word_map_dict

A dictionary with word mapped categoricals. Keys are items in the datafile, values are what they will be mapped to through the word mapping file. Unmapped items are also added as key, value pair.

Returns:dict.

WordMapping

class tmtk.clinical.WordMapping(params=None)

Bases: tmtk.utils.filebase.FileBase

Class representing the word mapping file.

build_index(df=None)

Build and sort multi-index for dataframe based on filename and column number columns. If no df parameter is not set, build index for self.df.

Parameters:df – pd.DataFrame. Returns:pd.DataFrame.

create_df()

Create pd.DataFrame with a correct header.

Returns:pd.DataFrame.

get_word_map(var_id)

Return dict with value in data file, and the mapped value as keyword-value pairs.

Parameters:var_id – tuple of filename and column number. Returns:dict.

validate(verbosity=2)

word_map_changes(silent=False)

Determine changes made to word mapping file.

Parameters:silent – if True, only print output. Returns:if silent=False return dictionary with changes since load.

word_map_dicts

Dictionary with all variable ids as keys and word map dicts as value.

Annotations

Annotations Container

class tmtk.annotation.Annotations.Annotations(params_list=None, parent=None)

Bases: object

Class containing all AnnotationFile objects.

validate_all(verbosity=3)

Base class: AnnotationBase

class tmtk.annotation.AnnotationBase.AnnotationBase(params=None, path=None)

Bases: tmtk.utils.filebase.FileBase

Base class for annotation files.

marker_type

validate(verbosity=2)

ChromosomalRegions

class tmtk.annotation.ChromosomalRegions.ChromosomalRegions(params=None, path=None)

Bases: tmtk.annotation.AnnotationBase.AnnotationBase

Subclass for CNV (aCGh, qDNAseq) annotation

biomarkers

MicroarrayAnnotation

class tmtk.annotation.MicroarrayAnnotation.MicroarrayAnnotation(params=None, path=None)

Bases: tmtk.annotation.AnnotationBase.AnnotationBase

Subclass for microarray (mRNA) expression annotation files.

biomarkers

MirnaAnnotation

class tmtk.annotation.MirnaAnnotation.MirnaAnnotation(params=None, path=None)

Bases: tmtk.annotation.AnnotationBase.AnnotationBase

Subclass for micro RNA (miRNA) expression annotation files.

biomarkers

ProteomicsAnnotation

class tmtk.annotation.ProteomicsAnnotation.ProteomicsAnnotation(params=None, path=None)

Bases: tmtk.annotation.AnnotationBase.AnnotationBase

Subclass for proteomics annotation

biomarkers

High Dimensional data

HighDim

class tmtk.highdim.HighDim.HighDim(params_list=None, parent=None)

Bases: object

Container class for all High Dimensional data types.

Parameters:params_list – contains a list with Params objects.

high_dim_nodes

update_high_dim_paths(high_dim_paths)

Update sample mapping if path has been changed.

Parameters:high_dim_paths – dictionary with paths and old concept paths.

validate_all(verbosity=3)

HighDimBase

class tmtk.highdim.HighDimBase.HighDimBase(params=None, path=None, parent=None)

Bases: tmtk.utils.filebase.FileBase

Base class for high dimensional data structures.

validate(verbosity=3)

Validate high dimensional data object

Parameters:verbosity – set the verbosity of output, pick 0, 1, 2, 3 or 4. Returns:True if everything is okay, else return False.

CopyNumberVariation

class tmtk.highdim.CopyNumberVariation.CopyNumberVariation(params=None, path=None, parent=None)

Bases: tmtk.highdim.HighDimBase.HighDimBase

Base class for copy number variation datatypes (aCGH, qDNAseq)

allowed_header

remap_to(destination=None)

Parameters:destination – Returns:

samples

Expression

class tmtk.highdim.Expression.Expression(params=None, path=None, parent=None)

Bases: tmtk.highdim.HighDimBase.HighDimBase

Base class for microarray mRNA expression data.

samples

Mirna

class tmtk.highdim.Mirna.Mirna(params=None, path=None, parent=None)

Bases: tmtk.highdim.HighDimBase.HighDimBase

Base class for proteomics data.

samples

Proteomics

class tmtk.highdim.Proteomics.Proteomics(params=None, path=None, parent=None)

Bases: tmtk.highdim.HighDimBase.HighDimBase

Base class for proteomics data.

samples

ReadCounts

class tmtk.highdim.ReadCounts.ReadCounts(params=None, path=None, parent=None)

Bases: tmtk.highdim.HighDimBase.HighDimBase

Subclass for ReadCounts.

allowed_header

remap_to(destination=None)

Parameters:destination – Returns:

samples

SampleMapping

class tmtk.highdim.SampleMapping.SampleMapping(path=None)

Bases: tmtk.utils.filebase.FileBase

Base class for subject sample mapping

get_concept_paths

Get all concept paths from file, replaces ATTR1 and ATTR2.

Returns:dictionary with md5 hash values as key and paths as value

platform

Returns:the platform id in this sample mapping file.

samples

slice_path(path)

Give slice of the dataframe where the paths are equal to given path. :param path: path (will be converted using global logic). :return: slice of dataframe.

study_id

Returns:study_id in sample mapping file

update_concept_paths(path_dict)

validate(verbosity=2)

Checks whether transmart-batch likes this file.

Metadata Tags

Tags

class tmtk.tags.Tags.MetaDataTags(params=None, parent=None)

Bases: tmtk.utils.filebase.FileBase

static create_df()

get_tags()

generator that gets tags from tags file.

Returns:tuples (<path>, <title>, <description>)

invalid_paths

tag_paths

Return tag paths delimited by the path_converter.

validate(verbosity=2)

Utilities

FileBase

class tmtk.utils.filebase.FileBase

Bases: object

Super class with shared utilities for file objects.

df

The pd.DataFrame for this file object.

df_has_changed

header

name

save()

Overwrite the original file with the current dataframe.

tabs_in_first_line()

Check if file is tab delimited.

write_to(path, overwrite=False)

Wrapper for tmtk.utils.df2file().

Parameters:

• path – path to write file to.
• overwrite – write over existing files in the filesystem)

Generic module

tmtk.utils.Generic.clean_for_namespace(path) → str

Converts a path and returns a namespace safe variant. Converts characters that give errors to underscore.

Parameters:path – usually a descriptive subdirectory Returns:string

tmtk.utils.Generic.column_map_diff(a_column, b_column)

tmtk.utils.Generic.df2file(df=None, path=None, overwrite=False)

Write a dataframe to file safely. Does not overwrite existing files automatically. This function converts concept path delimiters.

Parameters:

• df – pd.DataFrame
• path – path to write to
• overwrite – False (default) or True

tmtk.utils.Generic.file2df(path=None)

Load a file specified by path into a Pandas dataframe. If hashed is True, return a a (dataframe, hash) value tuple.

Parameters:path – to file to load Returns:pd.DataFrame

tmtk.utils.Generic.find_fully_unique_columns(df)

Check if a dataframe contains a fully unique column (SUBJ_ID candidate).

Parameters:df – pd.DataFrame Returns:list of names of unique columns

tmtk.utils.Generic.fix_everything()

Scans over all the data and indicates which errors have been fixed. This function is great for stress relieve.

Returns:All your problems fixed by Rick

tmtk.utils.Generic.is_numeric(values)

Check if list of values are numeric.

Parameters:values – iterable

tmtk.utils.Generic.md5(s: str)

utf-8 encoded md5 hash string of input s.

Parameters:s – string Returns:md5 hash string

tmtk.utils.Generic.merge_two_dicts(x, y)

Given two dicts, merge them into a new dict as a shallow copy.

tmtk.utils.Generic.numeric(x)

tmtk.utils.Generic.path_converter(path, internal=False)

Convert paths by creating delimiters of backslash “” and “+” sign, additionally converting underscores “_” to a single space.

Parameters:

• path – concept path
• internal – if path is for internal use delimit with Mappings.PATH_DELIM

Returns:

delimited path

tmtk.utils.Generic.path_join(*args)

Join items with the used path delimiter.

Parameters:args – path items Returns:path as string

tmtk.utils.Generic.summarise(list_or_dict=None, max_items: int = 7) → str

Takes an iterable and returns a summarized string statement. Picks a random sample if number of items > max_items.

Parameters:

• list_or_dict – list or dict to summarise
• max_items – maximum number of items to keep.

Returns:

the items joined as string with end statement.

tmtk.utils.Generic.word_map_diff(a_word_map, b_word_map)

utils.CPrint module

class tmtk.utils.CPrint.CPrint

Bases: object

static error(message)

static head(message)

static info(message)

static okay(message)

static warn(message)

class tmtk.utils.CPrint.MessageCollector(verbosity=5)

Bases: object

error(message)

flush()

head(message)

info(message)

okay(message)

warn(message)

utils.Exceptions module

exception tmtk.utils.Exceptions.ClassError(found=None, expected=None)

Bases: BaseException

Error raised when unexpected class is found.

Parameters:

• found – is the Object class of found
• expected – is the required Object class

exception tmtk.utils.Exceptions.DatatypeError(found=None, expected=None)

Bases: BaseException

Error raised when incorrect datatype is found.

Parameters:

• found – is the datatype of object
• expected – is the required datatype

exception tmtk.utils.Exceptions.NotYetImplemented

Bases: BaseException

exception tmtk.utils.Exceptions.PathError(found=None)

Bases: BaseException

Error raised when an incorrect path is given.

exception tmtk.utils.Exceptions.TooManyValues(found=None, expected=None, id_=None)

Bases: BaseException

Error raised when too many values are found.

utils.HighDimUtils module

tmtk.utils.HighDimUtils.check_datafile_header_with_subjects(header_samples, mapping_samples)

Inputs two pandas series, returns all missing annotations.

Parameters:

• header_samples – samples from datafile header.
• mapping_samples – samples present in sample mapping file.

Returns:

a dict with mismapped and excluded samples.

tmtk.utils.HighDimUtils.find_missing_annotations(annotation_series, data_series)

Inputs two pandas series, returns all missing annotations.

Parameters:

• annotation_series – the annotation that will be uploaded.
• data_series – series of genes that is in the datafile.

Returns:

a new series with all missing annotations.

utils.mappings module

class tmtk.utils.mappings.Mappings

Bases: object

Collection of statics used in various parts of the code.

EXT_PATH_DELIM = '\\'

PATH_DELIM = '∕'

annotation_data_types = {'proteomics': 'Proteomics data (mass spec)', 'vcf': 'Genomic variant data', 'cnv': 'ACGH data', 'mirna': 'micro RNA data (PCR)', 'rnaseq': 'Messenger RNA data (sequencing)', 'expression': 'Messenger RNA data (microarray)'}

annotation_marker_types = {'mrna_annotation': 'Gene expression', 'cnv_annotation': 'Chromosomal', 'proteomics_annotation': 'PROTEOMICS', 'vcf_annotation': 'VCF', 'mirna_annotation': 'MIRNA_QPCR', 'rnaseq_annotation': 'RNASEQ_RCNT'}

cat_cd = 'Category Code'

cat_cd_s = 'ccd'

col_num = 'Column Number'

col_num_s = 'col'

column_mapping_header = ['Filename', 'Category Code', 'Column Number', 'Data Label', 'Magic 5th', 'Magic 6th', 'Concept Type']

column_mapping_s = ['fn', 'ccd', 'col', 'dl', 'm5', 'm6', 'cty']

concept_type = 'Concept Type'

concept_type_s = 'cty'

data_label = 'Data Label'

data_label_s = 'dl'

df_value = 'Datafile Value'

df_value_s = 'dfv'

filename = 'Filename'

filename_s = 'fn'

static get_annotations(dtype=None)

Return mapping for annotations classes. Return only for datatype if dtype is set. Else return full map.

Parameters:dtype – optional datatype (e.g. cnv_annotation) Returns:dict with mapping, or class.

static get_highdim(dtype=None)

Return mapping for high dimensional classes. Return only for datatype if dtype is set. Else return full map.

Parameters:dtype – optional datatype (e.g. cnv) Returns:dict with mapping, or class.

static get_params(dtype=None)

Return mapping for params classes. Return only for datatype if dtype is set. Else return full map.

Parameters:dtype – optional datatype (e.g. cnv) Returns:dict with mapping, or class.

magic_5 = 'Magic 5th'

magic_5_s = 'm5'

magic_6 = 'Magic 6th'

magic_6_s = 'm6'

map_value = 'Mapping Value'

map_value_s = 'map'

tags_description = 'Description'

tags_header = ['Concept Path', 'Title', 'Description', 'Weight']

tags_node_name = 'Tags'

tags_path = 'Concept Path'

tags_title = 'Title'

tags_weight = 'Weight'

word_mapping_header = ['Filename', 'Column Number', 'Datafile Value', 'Mapping Value']

Toolbox package

Generate chromosomal regions file

tmtk.toolbox.generate_chromosomal_regions_file.generate_chromosomal_regions_file(platform_id=None, reference_build='hg19', **kwargs)

This creates a new chromosomal regions annotation file.

Parameters:

• platform_id – Give the new platform a name to fill first column
• reference_build – choose either hg18, hg19 or hg38

Returns:

a pandas dataframe with the new platform

Remap chromosomal regions data

tmtk.toolbox.remap_chromosomal_regions.map_index_to_region_ids(gene, origin_platform, region_origin)

tmtk.toolbox.remap_chromosomal_regions.remap_chromosomal_regions(origin_platform=None, destination_platform=None, datafile=None, flag_indicator='.flag', to_dest=2, start_dest=3, end_dest=4, region_dest=1, chr_origin=2, start_origin=3, end_origin=4, region_origin=1, region_data=0)

tmtk.toolbox.remap_chromosomal_regions.return_mean(datafile, mapping, flag_columns=None)

Study Wizard

tmtk.toolbox.wizard.create_study(path)

Start a study object by pointing by giving a folder that contains clinical data files only.

Parameters:path – path to folder with files. Returns:study object.

The Arborist

tmtk.arborist.common module

tmtk.arborist.common.call_boris(to_be_shuffled=None, **kwargs)

This function loads the Arborist if it has been properly installed in your environment.

Parameters:to_be_shuffled – has to be either a tmtk.Study object, a Pandas column mapping dataframe, or a path to column mapping file.

tmtk.arborist.common.get_open_port()

Open an available port (as given by the OS) and return it.

Returns:the port number

tmtk.arborist.common.launch_arborist_gui(json_data: str, height=650)

Parameters:

• json_data –
• height –

Returns:

tmtk.arborist.common.update_clinical_from_json(clinical, json_data)

Parameters:

• clinical –
• json_data –

Returns:

tmtk.arborist.common.update_study_from_json(study, json_data)

Parameters:

• study –
• json_data –

Returns:

tmtk.arborist.connect_to_baas module

tmtk.arborist.connect_to_baas.get_instance_url(url)

tmtk.arborist.connect_to_baas.get_json_from_baas(url, username=None)

Get a json file from a Boris as a Service instance.

Parameters:

• url – url should study name and version. (e.g. http://transmart-arborist.thehyve.nl/trees/study-name/1/~edit/).
• username – if no username is given, you will be prompted for one.

Returns:

the JSON string from BaaS.

tmtk.arborist.connect_to_baas.json_url(url)

tmtk.arborist.connect_to_baas.login_url(url)

tmtk.arborist.connect_to_baas.publish_to_baas(url, json, study_name, username=None)

Publishes a tree on a Boris as a Service instance.

Parameters:

• url – url to a BaaS instance.
• json – the stringified json you want to publish.
• study_name – a nice name.
• username – if no username is given, you will be prompted for one.

Returns:

the url that points to the study you’ve just uploaded.

tmtk.arborist.connect_to_baas.start_session(url, username)

tmtk.arborist.flask_connection module

tmtk.arborist.flask_connection.single_forward_slashed(string)

Converts string so that all double and backslashes for a single forward slash.

Parameters:string – input string Returns:returns the new string

tmtk.arborist.flask_connection.urlencode_filter(s)

Necessary addition to Jinja2 filters, to escape chars for in url.

Parameters:s – Returns:

tmtk.arborist.jstreecontrol module

class tmtk.arborist.jstreecontrol.ConceptNode(path, var_id=None, node_type='numeric', data_args=None)

Bases: object

class tmtk.arborist.jstreecontrol.ConceptTree(json_data=None)

Bases: object

Build a ConceptTree to be used in the graphical tree editor.

add_node(path, var_id=None, node_type=None, data_args=None)

Add ConceptNode object nodes list.

Parameters:

• path – Concept path for this node.
• var_id – Unique ID that allows to keep track of a node.
• node_type – Explicitly set node type (highdim, numerical, categorical)
• data_args – Any additional parameters are put a ‘data’ dictionary.

column_mapping_file

Returns:Column Mapping file based on ConceptTree object.

high_dim_paths

All high dimensional nodes in concept tree as dict

jstree

tags_file

word_mapping

class tmtk.arborist.jstreecontrol.JSNode(path, oid=None, **kwargs)

Bases: object

This class exists as a helper to the JSTree. Its “json_data” method can generate sub-tree JSON without putting the logic directly into the JSTree.

get_child(var_id, text)

json_data()

class tmtk.arborist.jstreecontrol.JSTree(concept_nodes)

Bases: object

An json like object that converts a list of nodes into something that jQuery jstree can use.

json_data

Convert this object to json ready to be consumed by jstree.

json_data_string

Returns:Returns the json_data properly formatted as string.

pretty(root=None, depth=0, spacing=2)

Create a pretty representation of tree.

to_clipboard()

class tmtk.arborist.jstreecontrol.MyEncoder(skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: json.encoder.JSONEncoder

Overwriting the standard JSON Encoder to treat numpy ints as native ints.

default(obj)

tmtk.arborist.jstreecontrol.create_concept_tree(column_object)

Parameters:column_object – tmtk.Study object, tmtk.Clinical object, or ColumnMapping dataframe Returns:json string to be interpreted by the JSTree

tmtk.arborist.jstreecontrol.create_tree_from_clinical(clinical_object, concept_tree=None)

Parameters:

• clinical_object –
• concept_tree –

Returns:

tmtk.arborist.jstreecontrol.create_tree_from_df(df, concept_tree=None)

Parameters:

• df –
• concept_tree –

Returns:

tmtk.arborist.jstreecontrol.create_tree_from_study(study, concept_tree=None)

Parameters:

• study –
• concept_tree –

Returns:

tmtk.arborist.jstreecontrol.get_concept_node_from_df(x)

This is only used when a the arborist is called from a single DF.

Parameters:x – row in column mapping dataframe Returns:

Indices and tables

• Index
• Module Index
• Search Page