Marked point process variational autoencoder with applications to unsorted spiking activities

Spike train modeling across large neural populations is a powerful tool for understanding how neurons code information in a coordinated manner. Recent studies have employed marked point processes in neural population modeling. The marked point process is a stochastic process that generates a sequence of events with marks. Spike train models based on such processes use the waveform features of spikes as marks and express the generative structure of the unsorted spikes without applying spike sorting. In such modeling, the goal is to estimate the joint mark intensity that describes how observed covariates or hidden states (e.g., animal behaviors, animal internal states, and experimental conditions) influence unsorted spikes. A major issue with this approach is that existing joint mark intensity models are not designed to capture high-dimensional and highly nonlinear observations. To address this limitation, we propose a new joint mark intensity model based on a variational autoencoder, capable of representing the dependency structure of unsorted spikes on observed covariates or hidden states in a data-driven manner. Our model defines the joint mark intensity as a latent variable model, where a neural network decoder transforms a shared latent variable into states and marks. With our model, we derive a new log-likelihood lower bound by exploiting the variational evidence lower bound and upper bound (e.g., the χ upper bound) and use this new lower bound for parameter estimation. To demonstrate the strength of this approach, we integrate our model into a state space model with a nonlinear embedding to capture the hidden state dynamics underlying the observed covariates and unsorted spikes. This enables us to reconstruct covariates from unsorted spikes, known as neural decoding. Our model achieves superior performance in prediction and decoding tasks for synthetic data and the spiking activities of place cells.Author summary: Recent neural recording techniques provide simultaneous insights into large neural population activities. This type of data may offer new insights into the mechanisms of high-dimensional and highly nonlinear neural coding, which are not available with small data. The marked point process is a promising tool for modeling neural population activities. This model avoids the preprocessing of spike assignment and directly estimates the generative distribution of neural recordings. However, when analyzing high-dimensional data or nonlinear dynamics underlying the data, developing appropriate parametric models for the marked point process becomes challenging. To address this limitation, we propose a new marked point process model that uses a variational autoencoder. By utilizing the strengths of neural networks, our model automatically estimates the spike generation distribution from neural recordings, enabling us to find its dependence on animal behavior or external stimuli. Our model is also capable of performing neural decoding, a fundamental technique used to reconstruct variables in the outside world from neural population activities. By integrating our model with neural network-based state space models, we can reconstruct animal behavior or external stimuli from unsorted spikes in a data-driven manner. This could help researchers formulate hypotheses regarding the connection between neural population activities and other influencing factors, and design further experiments and analyses.