Storing supplemental data on ``Study`` objects
==============================================

A recently added feature is the ability to store any arbitrary pandas
dataframe on ``study.supplemental``, and this will get re-loaded every
time you ``embark`` on that datapackage. Let's start with the
batch-corrected `BrainSpan <http://www.brainspan.org/>`_ Allen Brain
Institute's Brain Atlas data.

.. code:: python

    import flotilla
    study = flotilla.embark(flotilla._brainspan)

.. parsed-literal::

    Creating a directory for saving your flotilla projects: /home/travis/flotilla_projects
    Creating a directory for saving the data for this project: /home/travis/flotilla_projects/brainspan_filtered_and_markers_amazon
    https://s3-us-west-2.amazonaws.com/flotilla/brainspan_batch_corrected_for_amazon_s3/datapackage.json has not been downloaded before.
    	Downloading now to /home/travis/flotilla_projects/brainspan_filtered_and_markers_amazon/datapackage.json
    2015-06-09 22:42:57	Parsing datapackage to create a Study object
    https://s3-us-west-2.amazonaws.com/flotilla/brainspan_batch_corrected_for_amazon_s3/expression_feature.csv has not been downloaded before.
    	Downloading now to /home/travis/flotilla_projects/brainspan_filtered_and_markers_amazon/expression_feature.csv
    https://s3-us-west-2.amazonaws.com/flotilla/brainspan_batch_corrected_for_amazon_s3/expression.csv has not been downloaded before.
    	Downloading now to /home/travis/flotilla_projects/brainspan_filtered_and_markers_amazon/expression.csv
    https://s3-us-west-2.amazonaws.com/flotilla/brainspan_batch_corrected_for_amazon_s3/metadata.csv has not been downloaded before.
    	Downloading now to /home/travis/flotilla_projects/brainspan_filtered_and_markers_amazon/metadata.csv
    2015-06-09 22:43:12	Initializing Study
    2015-06-09 22:43:12	Initializing Predictor configuration manager for Study
    2015-06-09 22:43:12	Predictor ExtraTreesClassifier is of type <class 'sklearn.ensemble.forest.ExtraTreesClassifier'>
    2015-06-09 22:43:12	Added ExtraTreesClassifier to default predictors
    2015-06-09 22:43:12	Predictor ExtraTreesRegressor is of type <class 'sklearn.ensemble.forest.ExtraTreesRegressor'>
    2015-06-09 22:43:12	Added ExtraTreesRegressor to default predictors
    2015-06-09 22:43:12	Predictor GradientBoostingClassifier is of type <class 'sklearn.ensemble.gradient_boosting.GradientBoostingClassifier'>
    2015-06-09 22:43:12	Added GradientBoostingClassifier to default predictors
    2015-06-09 22:43:12	Predictor GradientBoostingRegressor is of type <class 'sklearn.ensemble.gradient_boosting.GradientBoostingRegressor'>
    2015-06-09 22:43:12	Added GradientBoostingRegressor to default predictors
    2015-06-09 22:43:12	Loading metadata
    2015-06-09 22:43:12	Loading expression data
    2015-06-09 22:43:12	Initializing expression
    2015-06-09 22:43:13	Done initializing expression
    2015-06-09 22:43:16	Successfully initialized a Study object!


Let's take a look at how big this expression matrix is.

.. code:: python

    study.expression.data.shape



.. parsed-literal::

    (493, 14321)



Yikes, 14,321 features is a lot! Let's subset on just the most variant
genes. By default, this is the genes whose variance is two standard
deviations away from the mean variance of all genes.

.. code:: python

    variant_ids = study.expression.feature_subsets['variant']
    variant_expression = study.expression.data.ix[:, variant_ids]
    variant_expression.shape



.. parsed-literal::

    (493, 553)



564 features isn't so bad. Let's correlate all features to each other in
this subset.

.. code:: python

    %%time
    variant_expression_corr = variant_expression.corr()
    variant_expression_corr.head()

.. parsed-literal::

    CPU times: user 393 ms, sys: 0 ns, total: 393 ms
    Wall time: 392 ms


That didn't take *too* long, but I'm sure you can imagine it would take
a really long time for ALL genes!

Now let's assign this to the ``study.supplemental`` object with a name
of our choice. To keep things simple, I'm gonna give it the same name.

.. code:: python

    study.supplemental.variant_expression_corr = variant_expression_corr
Now let's save the object and re-``embark`` to make sure it's there.

.. code:: python

    study.save('brainspan2')
    study2 = flotilla.embark('brainspan2')

.. parsed-literal::

    Wrote datapackage to /home/travis/flotilla_projects/brainspan2/datapackage.json
    2015-06-09 22:44:04	Reading datapackage from /home/travis/flotilla_projects/brainspan2/datapackage.json
    2015-06-09 22:44:04	Parsing datapackage to create a Study object
    2015-06-09 22:44:10	Initializing Study
    2015-06-09 22:44:10	Initializing Predictor configuration manager for Study
    2015-06-09 22:44:10	Predictor ExtraTreesClassifier is of type <class 'sklearn.ensemble.forest.ExtraTreesClassifier'>
    2015-06-09 22:44:10	Added ExtraTreesClassifier to default predictors
    2015-06-09 22:44:10	Predictor ExtraTreesRegressor is of type <class 'sklearn.ensemble.forest.ExtraTreesRegressor'>
    2015-06-09 22:44:10	Added ExtraTreesRegressor to default predictors
    2015-06-09 22:44:10	Predictor GradientBoostingClassifier is of type <class 'sklearn.ensemble.gradient_boosting.GradientBoostingClassifier'>
    2015-06-09 22:44:10	Added GradientBoostingClassifier to default predictors
    2015-06-09 22:44:10	Predictor GradientBoostingRegressor is of type <class 'sklearn.ensemble.gradient_boosting.GradientBoostingRegressor'>
    2015-06-09 22:44:10	Added GradientBoostingRegressor to default predictors
    2015-06-09 22:44:10	Loading metadata
    2015-06-09 22:44:11	Loading expression data
    2015-06-09 22:44:11	Initializing expression
    2015-06-09 22:44:13	Done initializing expression
    2015-06-09 22:44:16	Successfully initialized a Study object!


Let's make sure our ``variant_expression_corr`` dataframe is really
there.

.. code:: python

    study2.supplemental.variant_expression_corr.head()



.. raw:: html

    <div style="max-height:1000px;max-width:1500px;overflow:auto;">
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>ENSG00000003137</th>
          <th>ENSG00000004848</th>
          <th>ENSG00000006016</th>
          <th>ENSG00000006116</th>
          <th>ENSG00000006128</th>
          <th>ENSG00000006377</th>
          <th>ENSG00000007350</th>
          <th>ENSG00000016082</th>
          <th>ENSG00000041353</th>
          <th>ENSG00000041982</th>
          <th>...</th>
          <th>ENSG00000258283</th>
          <th>ENSG00000258403</th>
          <th>ENSG00000258444</th>
          <th>ENSG00000258518</th>
          <th>ENSG00000258752</th>
          <th>ENSG00000259190</th>
          <th>ENSG00000259279</th>
          <th>ENSG00000259373</th>
          <th>ENSG00000259410</th>
          <th>ENSG00000259603</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>ENSG00000003137</th>
          <td>1.000000</td>
          <td>-0.046835</td>
          <td>0.090661</td>
          <td>0.053573</td>
          <td>-0.047665</td>
          <td>-0.155271</td>
          <td>0.054222</td>
          <td>-0.160111</td>
          <td>-0.115487</td>
          <td>-0.044074</td>
          <td>...</td>
          <td>-0.224868</td>
          <td>-0.019139</td>
          <td>-0.121898</td>
          <td>-0.392903</td>
          <td>-0.122529</td>
          <td>0.273103</td>
          <td>0.136641</td>
          <td>0.448154</td>
          <td>0.445432</td>
          <td>0.041884</td>
        </tr>
        <tr>
          <th>ENSG00000004848</th>
          <td>-0.046835</td>
          <td>1.000000</td>
          <td>0.612271</td>
          <td>0.707699</td>
          <td>0.558755</td>
          <td>0.671949</td>
          <td>0.028770</td>
          <td>0.148974</td>
          <td>0.448605</td>
          <td>0.001084</td>
          <td>...</td>
          <td>0.701812</td>
          <td>0.328554</td>
          <td>0.040970</td>
          <td>-0.478688</td>
          <td>-0.082581</td>
          <td>-0.196485</td>
          <td>0.135765</td>
          <td>-0.662072</td>
          <td>-0.570054</td>
          <td>-0.623628</td>
        </tr>
        <tr>
          <th>ENSG00000006016</th>
          <td>0.090661</td>
          <td>0.612271</td>
          <td>1.000000</td>
          <td>0.652296</td>
          <td>0.585003</td>
          <td>0.450687</td>
          <td>0.037159</td>
          <td>0.111290</td>
          <td>0.414487</td>
          <td>0.022681</td>
          <td>...</td>
          <td>0.650395</td>
          <td>0.300988</td>
          <td>0.233828</td>
          <td>-0.391661</td>
          <td>0.055337</td>
          <td>-0.289575</td>
          <td>0.060468</td>
          <td>-0.418865</td>
          <td>-0.398856</td>
          <td>-0.348691</td>
        </tr>
        <tr>
          <th>ENSG00000006116</th>
          <td>0.053573</td>
          <td>0.707699</td>
          <td>0.652296</td>
          <td>1.000000</td>
          <td>0.516889</td>
          <td>0.424020</td>
          <td>-0.185337</td>
          <td>-0.071044</td>
          <td>0.469453</td>
          <td>-0.299232</td>
          <td>...</td>
          <td>0.687966</td>
          <td>0.526633</td>
          <td>-0.027971</td>
          <td>-0.569975</td>
          <td>-0.209932</td>
          <td>-0.020450</td>
          <td>0.220832</td>
          <td>-0.560163</td>
          <td>-0.465723</td>
          <td>-0.655024</td>
        </tr>
        <tr>
          <th>ENSG00000006128</th>
          <td>-0.047665</td>
          <td>0.558755</td>
          <td>0.585003</td>
          <td>0.516889</td>
          <td>1.000000</td>
          <td>0.715297</td>
          <td>0.000925</td>
          <td>0.415798</td>
          <td>0.564539</td>
          <td>0.053909</td>
          <td>...</td>
          <td>0.716306</td>
          <td>0.429350</td>
          <td>0.264993</td>
          <td>-0.256103</td>
          <td>0.155573</td>
          <td>-0.431776</td>
          <td>-0.079680</td>
          <td>-0.378022</td>
          <td>-0.389113</td>
          <td>-0.255839</td>
        </tr>
      </tbody>
    </table>
    <p>5 rows × 553 columns</p>
    </div>



Yay, it's here!