ML serialization framework

[JonathanChiang]

Hi,

I was curious if there was any concrete roadmap to build compatibility between these Apache frameworks.

Would love to do data processing in python/R in arrow for deep learning or tabular data model building.

My use case is model building and interoperability between R and Python users at Stanford Biomedical Informatics.

My dream would be to have cloud based /local hybrid deep learning and reduce the amount serialization needed between R and Python users.

Thanks,
Jonathan

hi Jonathan,

Well, one of the good things about Apache projects is that working in
private is frowned upon, so if you don't find anything in JIRA or in
the mailing list archives then the answer is most probably "no" :)

That being said I'm interested in integrations between Apache Arrow
and machine learning frameworks. Generally some serialization is
necessary at some point because machine learning frameworks operation
in many cases on (homogeneously-typed) multidimensional arrays or
sparse matrices

Wes Mckinney

[JonathanChiang]

##WES MCKINNEY

hey Jonathan,

On Thu, Jun 13, 2019 at 9:52 AM Jonathan Chiang chiang810@gmail.com wrote:

Hi Wes,

My name is Jonathan, and we've interacted a bit before about machine learning serialization, and cloud implementation. Prior to this year, I was dead set on a PhD. However, what I truly value was seeing the value of my ideas come to life.

I'm still very passionate about model and data serialization applied to distributed machine learning and deep learning. I'm actually collaborating with a group from Mass General Hospital, Stanford and MIT.

I wanted to know from you, at what point did you take the leap to make pandas? Your work is very inspiring to me, and the idea that you would risk your academic and industry career is rare and awesome.

Thanks!

Well, when I started working on it I wasn't taking that much risk, it
was in the context of my day job. I did go out on a limb in 2011 to
spend a lot more unpaid time on it. I discussed the process in this
talk from 2015

https://www.slideshare.net/wesm/my-data-journey-with-python

I had the benefit of some personal savings, no student loan debt, and
financially secure family (in the event that I went broke) working in
my favor. I wouldn't recommend it to everyone but I didn't find the
"safe" career path very motivating.

I appreciate your time and your willingness to double down on your dreams with Apache Arrow.

No worries!

best of luck