Sergey Karayev | Blog

Setting up a development environment on Mac OS X 10.8 Mountain Lion

2012-08-08T00:00:00-07:00

Do you want a good modern development setup? Ruby and Node for all the web goodness and Python with a beautiful ipython console, Numpy, and Scipy for math and statistical computing?

This is the gospel. This is what you will do.

Building blocks: Homebrew, XCode, X11, and Git

This guide is for starting from a fresh Mountain Lion install, with nothing else installed. You can approximate that state, by getting rid of all your macports and finks and what have you. Delete your /usr/local. Uninstall all XCodes are their developer tools.

Install homebrew:

ruby <(curl -fsSk https://raw.github.com/mxcl/homebrew/go)

Homebrew lets us effortlessly install things from source. It is good, but it needs some help: Mountain Lion doesn’t come with developer tools, and homebrew is not able to do much right now.

Go to the App Store and download XCode (I am writing this in early August 2012, and the current version is 4.4). This can take a little bit, so while it’s downloading, let’s install X11 libraries that Mountain Lion stripped out.

Go here and download 2.7.2+. Install it, and after it’s done, fix the symlink it makes:

ln -s /opt/X11 /usr/X11

When XCode download is done, launch it and go to Preferences, Downloads tab, and install the Command Line Tools. When it finishes, we are almost ready to brew. Before we do, let’s have XCode tell everyone where the tools are (this tip is from Get Mountain Lion and Homebrew to Be Happy).

sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer

Open up a new shell to make sure everything is loaded from scratch, and check that homebrew is good to go:

brew doctor

Fix the stuff it complains about until it doesn’t.

Now let’s get git:

brew install git

And we are off to the races!

Science stuff: Python and the SciPy stack

We are going to take advantage of a nice man’s labor of love—the Scipy Superpack—to dramatically cut down the time and effort it will take us to get from nothing to a full Matlab and R replacement.

To use it most effectively, we are going to base everything on the system Python, which right now is version 2.7.2. This is 0.0.1 versions behind the latest and greatest, but I think we’ll survive.

Python is best installed through an environment manager.

easy_install pip
pip install virtualenv
mkdir ~/.virtual_envs
virtualenv ~/.virtual_envs/system
source ~/.virtual_envs/system/bin/activate

This downloads a package manager, installs virtualenv, duplicates the system environment to a directory in your home directory, and activates this environment.

Doing which python should now show a .virtual_envs-containing path. Make sure you add the last line to your ~/.bashrc.

Now install the superpack. Since we may want to keep up to date on the exciting scientific python developments, let’s check out the git repository so that we can update faster in the future.

mkdir ~/local && cd ~/local
git clone git://github.com/fonnesbeck/ScipySuperpack.git
cd ScipySuperpack
sh install_superpack.sh

Select y at the prompt, and that’s it! The script will install gfortran and binary builds of the latest development versions of Numpy, Scipy, Matplotlib, IPython, Pandas, Statsmodels, Scikit-learn, and PyMC, as well as their dependencies.

Now let’s get IPython to look beautiful using qtconsole. Download Qt 4.7.4 libraries and PySide libraries.

Unfortunately, the PySide package installs its stuff into the system python site-packages directory, and our virtualenv ipython doesn’t see it. We could try building PySide from source, but instead we are just going to symlink the relevant stuff from the system to our virtualenv folder. This isn’t very clean, but it works for me.

ln -s /Library/Python/2.7/site-packages/pysideuic $HOME/.virtual_envs/system/lib/python2.7/site-packages
ln -s /Library/Python/2.7/site-packages/PySide $HOME/.virtual_envs/system/lib/python2.7/site-packages

Install some remaining dependencies (for some reason, the DateUtils package that the Superpack installs doesn’t work right for me):

pip install pygments
pip install dateutils

Now try it out:

ipython qtconsole --pylab=inline

In the qtconsole, try

plot(randn(500),rand(500),'o',alpha=0.2)

And enjoy the inline goodness.

Web stuff: Ruby, Node

Ruby is also best installed using an environment manager. Install RVM:

curl -L https://get.rvm.io | bash -s stable

Open up a new shell and let RVM install itself into your bashrc:

source ~/.rvm/scripts/rvm

Open up another shell and test that it works:

type rvm | head -n 1

should give rvm is a function. Great.

Now let’s install a version of Ruby. 1.9.3 (the latest version) works fine with the compilers provided by XCode 4.4, so:

rvm install 1.9.3
rvm use 1.9.3 --default

You should be all set now. For sanity, check that which bundle shows some .rvm-derived path. If there are problems, consult the detailed installation guide.

For Node and its package manager, simply

brew install node

Lastly, you may want to install Heroku at some point. I ran into a problem when installing from the Heroku Toolbelt package. Instead, simply

gem install heroku
gem install foreman

And you’re done. Good stuff.

CabFriendly -- a cloud-based mobile web app.

2011-12-13T00:00:00-08:00

(Joint work with Adam Roberts and Harold Pimentel.)

We have developed a cloud-based mobile web application to match users who request similar trips and would like to share a cab. The application is hosted on Amazon’s EC2 service and combines several open-source frameworks (Django, PostgresQL, Redis, Node.js) with social networking (Facebook), mapping, and location-awareness (Google) APIs. The modularity of our design allows the service to easily scale in the cloud as the user base grows.

Use it!

Architecture

Use case scenario

Attentional Object Detection -- introductory slides.

2011-03-24T00:00:00-07:00

For the Berkeley computer vision retreat, I made a little presentation outlining my case for using ideas in sequential decision making in object detection. It is meant partly to start conversation on the subject, and partly to summarize four interesting papers in this vein. Some notable things are omitted, such as region-based proposal approaches.

Attentional Object Detection - introductory slides.

View more presentations from sergeykarayev

Download pdf, or Keynote source.

Review of Kanan and Cottrell, Robust Classification of Objects, Faces, and Flowers Using Natural Image Statistics, CVPR 2010.

2011-01-24T00:00:00-08:00

Review of Kanan and Cottrell, Robust Classification of Objects, Faces, and Flowers Using Natural Image Statistics, CVPR 2010.

The paper’s approach has three parts. The first is using an ICA-based spatial pyramid feature; the second is computing a saliency map to sample interest points; and the third is in using Naive Bayes Nearest Neighbor (NBNN) for classification. The approach is evaluated on three single-object datasets: Caltech-101 and -256, Aleix and Robert faces dataset of 120 individuals with 26 images each, and 102 Flowers (8200 images). The results are best yet published for Caltech-101 single-feature approaches, and match best multiple-feature performances; comparable to state-of-the-art on Caltech-256; match state-of-the-art on the AR Faces; and beat the single previously published result on the Flowers dataset.

ICA-based Local Features and Saliency

The images are first pre-processed by converting to a standard size, converting to the LMS color space (designed to match human color receptor distributions), normalizing, and then applying a nonlinear transform inspired by modulation to luminance that happens in photoreceptors (a logarithmic compression). [Note: It would be interesting to see the effects of not performing this mapping.]

ICA filters of size $b \times b$ ($b$ tuned on a Butterfly and Bird dataset to 24 pixels) are learned on about 5000 color image patches from the McGill color image dataset. To learn $d$ ICA features, the authors first run PCA on the patches, discard the first principal component, retain $d$ following principal components, and then learn the ICA decomposition. I’m not quite sure how this works—I guess ICA is then only able to learn $d$ non-garbage bases?

Saliency Map

The ICA bases are used to place a saliency map over the image following the Saliency Using Natural statistics (SUN) framework \cite{Zhang:2008:SUN}. The basic idea is that saliency of a point is the inverse $P(F)^{-1}$ of its probability under the ICA model $P(F=\mathbf{f})=\prod_i P(\mathbf{f}_i)$. Each unidimensional distribution is fit with a generalized Gaussian distribution:
\[ P(\mathbf{f}_i) = \frac{\theta_i}{2 \sigma_i \Gamma(\theta_i^{-1})} exp(-|\frac{\mathbf{f}_i}{\sigma_i}|^{\theta_i}) \]
Parameters are fit still using the McGill color database. A further strange nonlinear weighting of the dimensions of $\mathbf{f}$ is then done to weight rarer responses more heavily.

Fixations

The saliency map is normalized to a probability distribution, and “fixations” are sampled from it $T$ times. At each location $l_t$, an interesting fixation feature is extracted. It is a spatial pyramid over an area of $w=51$ pixels, using average pooling. So, the initial window of $w \times w \times d$ is represented by a vector of size $21d$, where $21 = 4 \times 4 \times 2 \times 2 \times 1 \times 1$ shows the structure of the spatial pyramid. Importantly, the normalized location $l_t$ of the fixation is also stored. To cast SIFT in this framework, we would set $w=17$, $d=8$, and the spatial aggregation would be a flat $4 \times 4$ grid.

After gathering $T$ fixations on every image in the training set, the unit-normalized SP vectors are then additionally processed by retaining only the first 500 PCA components and whitening them. The chain of re-normalizations in this paper is quite long and I would appreciate theoretical justifications for these decisions.

Classification

The paper uses Kernel Density Estimation (KDE) to model $P(\mathbf{g}_t|C=k)$, where $\mathbf{g}_t$ is the vector of fixation features. A Naive Bayes assumption is made, such that each fixation contributes independently to the total probability. The posterior is estimated with Bayes rule, assuming uniform class priors. 1-nearest neighbor KDE is used, and the Euclidean distance between the fixation locations is considered in addition to the feature-to-exemplar distance. The final posterior probability is:
\[ P(\mathbf{g}_t | C=k) \propto max_i \frac{1}{||\mathbf{w}_{k,i}-\mathbf{g}_t||^2_2 + \alpha ||\mathbf{v}_{k,i}-\ell_t||^2_2 + \epsilon} \]
where $\mathbf{w}_{k,i}$ is a vector representing the $i$’th examplar of a fixation from class $k$.

Discussion

The authors attribute the strength of their approach largely to the exemplar-based classifier. Their approach does outperform the comparable single-descriptor version of the Boiman and Irani NBNN classifier \cite{Boiman:2008}; that could be due to a number of factors:

1. ~~They also use location information in their comparison of fixations.~~ EDIT: NBNN paper also appends location to the feature vector.
2. They sample features from a saliency map (vs. densely for NBNN)
3. They use their ICA feature instead of SIFT and other standard descriptors.

It would be excellent to see a controlled evaluation of each of these factors. The paper as it is presents a very specific and unorthodox approach, and does not justify many of its design decisions.

My questions:

1. What is the contribution of the saliency map? How would the performance change under a random sampling scheme? What about an interest-point sampling scheme?
2. Why is the saliency computed at a single scale? The only reason for this working well is that the dataset is single-object and fixed-scale.
3. How would performance change if a standard feature, for example SIFT, was extracted instead of the ICA SP feature?
4. What is the contribution of the location feature? Why is it weighted at $\alpha=0.5$; what would cross-validation tune it to?

In my mind, the most important part of the approach is NN classification. It would be interesting to re-implement this framework with a different bottom-up saliency map, for example the multi-scale one used in \cite{Alexe:2010} and traditional SIFT features.

Review of Itti and Koch, Computational Modeling of Visual Attention, Nature Neuroscience 2001.

2011-01-01T00:00:00-08:00

Review of Itti and Koch, Computational Modeling of Visual Attention, Nature Neuroscience 2001.

Why Attention?

The reigning paradigm for object detection is a scanning window approach, where object-specific windows are considered over different scales and locations of an image. This is an expensive and inelegant approach, and assuredly not how humans perform visual search. While of course we do not know how that is done, some observations can be grouped under a loose term of attention; for example, search time will be lowered if people are told a distinguishing characteristic, such as color, of an object they are looking for in a cluttered scene.

The main view of attention in this sense is by analogy to a spotlight. At least in our visual systems, not every part of a scene can be processed at a high level at once. Attention is the bottleneck through which high level processing flows. If we would use intuitions and experimental observations about visual attention in computational models of vision, we should focus on replacing exhaustive window search with a more intelligent spotlight.

Here I will summarize an influential synthesis of attentional models by Itti and Koch. In a follow-up post, I will go over a newly published work from Chikkerur et al. outlining a highly promising Bayesian model of attention.

The Review

The authors of this 2001 paper synthesize prior work on computational models of attentions and list five essential components of a model for bottom-up attention:

Perceptual saliency depends on surrounding context of stimuli. This is pre-attentive.
Bottom-up processing seems to culminate in a single “saliency map.”
Sequential nature of attention needs to be explained by something like inhibition of return.
Implicit (covert) and eye movement (overt) attentional deployments are coupled, posing coordinate system challenges to computational models.
Scene understanding and object recognition influence attention.

The authors first review the two-component framework for attention. Bottom-up attention is driven by saliency cues in the image and is on the order of 50ms. Top-down attention is driven by high-level cues and is on the order of 200ms, which is also around the time it takes to re-fixate the eyes.

Interesting fact: visual sensory input is estimated to be $10^7$-$10^8$ bits per second at the optic nerve. Authors make the bottleneck analogy: “Attention allows us to break down the problem of understanding a visual scene into a rapid series of computationally less demanding, localized visual analysis problems.”

I am not that interested in the brain areas involved, but a basic view never hurts. From the visual cortex, processing proceeds through the dorsal stream of the posterior parietal cortex and the ventral stream of the inferotemporal cortex. The prefrontal cortex is commonly viewed as the seat of attention. The processing streams converge there. Motor systems and high-level cognition are controlled by the PFC, including eye movement through the superior colliculus.

Pre-attentive computation of visual features

Early vision proceeds in parallel across the entire visual field. It is not purely feedforward. Attention can modulate early processing, virtually increasing the stimulus strength. Different features contribute with different weights. There is little evidence for cross-modality interactions at a given visual area, and we lack the ability to efficiently detect conjunctive targets.

Most importantly, feature contrast, not absolute strength, is what matters. Contrast extends past neuronal receptive fields (and contour completion could be due to long-range connections). My aside: our best current models of local feature descriptors attempt to achieve this effect, rather inelegantly.

Saliency map

Although there are multiple feature representation of the visual field, there is only one attentional focus. Most computational models hypothesize that the feature maps feed into a unique saliency map, which then controls attentional deployment. There is not a lot of biological evidence given for this hypothesis, but models with that assumption sometimes match human performance.

For example, the authors’ previously published computational model that uses non-classical surround modulation effects (as mentioned above) seems to reproduce human behavior in some visual search tasks, as well as some automatic salient object detection in real color images.

Attentional selection; interplay with eye movements

Given a saliency map, attention is deployed to the most salient point in the visual scene. There needs to be a mechanism for sequential selection of points of lower saliency, however—otherwise, attention would remain fixed. An efficient computational strategy is transient inhibition of the currently attended location, or inhibition of return (IOR).

The paper does not mention that the patterns of sequences of attentional deployments may carry information useful to object recognition, which is the hypothesis of a paper I plan on reviewing ¹.

IOR seems simple but shows complex behavior. For example, it seems to be object-bound, tracking moving objects and compensating for motion of the observer. More basically, covert attention must interplay with overt attention (eye movements), which poses challenges to the coordinate system for IOR.

On the subject of covert vs. overt attention, there is evidence that covert attention is deployed to the endpoint of an upcoming saccade.

Recognition

The previous four points describe a bottom-up attentional system, accounting for the first few hundred milliseconds after stimulus presentation. After that, top-down effects can strongly affect attentional deployment. The authors outline several models of top-down influence, focusing in particular on a model by Schill et al._ ², which will be subject of a later review. In a nutshell, the model’s assumption is that objects are recognized iteratively from coarse to fine, with eye movements that maximize information gain.

Another model that I will review is the “scanpath theory” ³ of Lawrence Stark, the late Berkeley optometry professor, who was a strong proponent of a prior-driven perceptual system. Eye movements over a scene, then, are mostly due to our cognitive model of what we expect to see.

¹ Paletta et al. Q-Learning of Sequential Attention for Visual Object Recognition from Informative Local Descriptors. ICML (2005)

² Schill et al. Scene analysis with saccadic eye movements: Top-down and bottom-up modeling. J. Electron. Imaging (2001) vol. 10 (1) pp. 152

³ Stark et al. Representation of human vision in the brain: How does human perception recognize images?. J. Electron. Imaging (2001) vol. 10 (1) pp. 123

Jupiter album art recreation

2010-08-16T00:00:00-07:00

I really like the cover of the album Jupiter by the band Starfucker (their music is awesome as well).

I thought it would look cool as a huge print on my wall. Unfortunately, I could not find a large enough image. So, I decided to recreate it using Processing. The basic idea was surprisingly easy to match. The hardest part to get close was the color distribution. After a little HSV space histogramming (suggested by Mario), I finally settled on manually picking a pallete of just six hues inspired by the original image, and then randomly (with some caveats) picking the saturation and brightness values. Pro tip: the slightly non-uniform widths of the stripes were modeled with a draw from the Dirichlet distribution.

The result looks pretty good to me, and can be generated at any resolution. Pictures of the HUGE poster forthcoming.

If your browser can handle HTML5 (most modern browsers can), check out an animated version, with a link to the source code.

Data collection photograph

2010-08-02T00:00:00-07:00

My friend and I rode bikes from Victoria, B.C. to San Francisco, along the Pacific coast. A little blog pictorially documented the trip.

2010-07-29T00:00:00-07:00