# Sifting Common Information from Many Variables

@inproceedings{Steeg2017SiftingCI, title={Sifting Common Information from Many Variables}, author={Greg Ver Steeg and Shuyang Gao and Kyle Reing and A. G. Galstyan}, booktitle={IJCAI}, year={2017} }

Measuring the relationship between any pair of variables is a rich and active area of research that is central to scientific practice. In contrast, characterizing the common information among any group of variables is typically a theoretical exercise with few practical methods for high-dimensional data. A promising solution would be a multivariate generalization of the famous Wyner common information, but this approach relies on solving an apparently intractable optimization problem. We… Expand

#### Supplemental Code

#### Paper Mentions

#### 7 Citations

The Information Sieve

- Computer Science, Mathematics
- ICML
- 2016

A new framework for unsupervised learning of representations based on a novel hierarchical decomposition of information is introduced and applied to a variety of fundamental tasks in unsuper supervised learning including independent component analysis, lossy and lossless compression, and predicting missing values in data. Expand

Algebraic properties of solutions to common information of Gaussian vectors under sparse graphical constraints

- Mathematics, Computer Science
- 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton)
- 2017

We formulate Wyner's common information for random vectors x ∊ Rn with joint Gaussian density. We show that finding the common information of Gaussian vectors is equivalent to maximizing a… Expand

Unsupervised Learning via Total Correlation Explanation

- Computer Science, Mathematics
- IJCAI
- 2017

The principle of Total Cor-relation Ex-planation (CorEx) is to learn representations of data that "explain" as much dependence in the data as possible to solve unsupervised learning problems across diverse domains including human behavior, biology, and language. Expand

Nonparanormal Information Estimation

- Mathematics, Computer Science
- ICML
- 2017

This work proposes estimators for mutual information when p is assumed to be a nonparanormal (a.k.a., Gaussian copula) model, a semiparametric compromise between Gaussian and nonparametric extremes, and shows these estimators strike a practical balance between robustness and scaling with dimensionality. Expand

Layered Synthesis of Latent Gaussian Trees

- Computer Science, Mathematics
- ArXiv
- 2016

It is shown that maximizing the achievable rate-region is equivalent to finding the worst case density for Bernoulli sign inputs where maximum amount of sign information is lost, and quantifies the amount of information loss due to unrecoverable sign information. Expand

PADDLE: Performance Analysis Using a Data-Driven Learning Environment

- Computer Science
- 2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
- 2018

This work proposes a unified machine learning framework, PADDLE, which is specifically designed for problems encountered during analysis of HPC data and can produce highly robust and interpretable models. Expand

Information Theoretic Study of Gaussian Graphical Models and Their Applications

- Computer Science
- 2017

The author’s aim is to provide a “roadmap” for the development of a coherent theory of randomness in trees based on topological and algebraic perspectives. Expand

#### References

SHOWING 1-10 OF 68 REFERENCES

Information Bottleneck for Gaussian Variables

- Mathematics, Computer Science
- J. Mach. Learn. Res.
- 2003

A formal definition of the general continuous IB problem is given and an analytic solution for the optimal representation for the important case of multivariate Gaussian variables is obtained, in terms of the eigenvalue spectrum. Expand

The Information Sieve

- Computer Science, Mathematics
- ICML
- 2016

A new framework for unsupervised learning of representations based on a novel hierarchical decomposition of information is introduced and applied to a variety of fundamental tasks in unsuper supervised learning including independent component analysis, lossy and lossless compression, and predicting missing values in data. Expand

Maximally Informative Hierarchical Representations of High-Dimensional Data

- Mathematics, Computer Science
- AISTATS
- 2015

A new approach to unsupervised learning of deep representations that is both principled and practical is established and demonstrates the usefulness of the approach on both synthetic and real-world data. Expand

Discovering Structure in High-Dimensional Data Through Correlation Explanation

- Mathematics, Computer Science
- NIPS
- 2014

It is demonstrated that Correlation Explanation automatically discovers meaningful structure for data from diverse sources including personality tests, DNA, and human language. Expand

Low Complexity Gaussian Latent Factor Models and a Blessing of Dimensionality

- Mathematics, Computer Science
- ArXiv
- 2017

This work designs a new approach to learning Gaussian latent factor models that benefits from dimensionality and relies on an unconstrained information-theoretic objective whose global optima correspond to structured latent factor generative models. Expand

A global geometric framework for nonlinear dimensionality reduction.

- Medicine
- Science
- 2000

An approach to solving dimensionality reduction problems that uses easily measured local metric information to learn the underlying global geometry of a data set and efficiently computes a globally optimal solution, and is guaranteed to converge asymptotically to the true structure. Expand

Representation Learning: A Review and New Perspectives

- Computer Science, Mathematics
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- 2013

Recent work in the area of unsupervised feature learning and deep learning is reviewed, covering advances in probabilistic models, autoencoders, manifold learning, and deep networks. Expand

NICE: Non-linear Independent Components Estimation

- Computer Science, Mathematics
- ICLR
- 2015

We propose a deep learning framework for modeling complex high-dimensional densities called Non-linear Independent Component Estimation (NICE). It is based on the idea that a good representation is… Expand

Independent component analysis: algorithms and applications

- Mathematics, Medicine
- Neural Networks
- 2000

The basic theory and applications of ICA are presented, and the goal is to find a linear representation of non-Gaussian data so that the components are statistically independent, or as independent as possible. Expand

Independent component analysis, A new concept?

- Mathematics, Computer Science
- Signal Process.
- 1994

An efficient algorithm is proposed, which allows the computation of the ICA of a data matrix within a polynomial time and may actually be seen as an extension of the principal component analysis (PCA). Expand