How to send XML Data to Database in Rails

rubyonrails-talk
1

I have attached a ruby program built to get the xml data out of files and display the text back to me on the screen. This program was built from a few different sources, as I'm still new with the Ruby language. But I have monkeyed around with it enough and have got it working how I want it.

What I'm trying to do is get this information into a database so I can search it using a web interface (through Rails of course).

Here is a sample of the data that is spit out to the screen when I run the program:

2

Matt Royer wrote in post #961585:

I have attached a ruby program built to get the xml data out of files and display the text back to me on the screen. This program was built from a few different sources, as I'm still new with the Ruby language. But I have monkeyed around with it enough and have got it working how I want it.

What I'm trying to do is get this information into a database so I can search it using a web interface (through Rails of course).

Here is a sample of the data that is spit out to the screen when I run the program:

----

//dc5/IP Office Calls/38/73/387398.xml <starttime>2009-05-05 15:28:47:000 -0600</starttime>

<endtime>2009-05-05 15:32:50:000 -0600</endtime>

<calldirection>Incoming</calldirection>

<filename>387398.wav</filename>

<recordingowner>5297</recordingowner>

<party id="1"> <number>5557915555</number> <name/> </party>

<party id="2"> <number>5297</number> <name>FredJones</name> </party>

----

What is the best way to go about getting this info loaded into a database? Should I created the table and fields first and then put all of the info in a yaml file and seed the database (I might have some questions on how to do that)? New xml files are getting dumped to these folders all the time, so I'm looking for a nice way to get this info into the database without too much hassle.

Then you don't want to seed the database -- seeding is only for relatively static data (such as state and country names) that needs to be loaded before you ever run the application. It sounds to me like you simply want to read the XML files and build INSERT queries. And unless I'm badly mistaken, you already know how to do that.

You may need a cron job or some such to parse new files as they come in.

I'm not sure how to make this data available to rails.

What's the problem, exactly? There's nothing here that you don't know.

Best,

3

Matt Royer wrote in post #961585:

I have attached a ruby program built to get the xml data out of files and display the text back to me on the screen. This program was built from a few different sources, as I'm still new with the Ruby language. But I have monkeyed around with it enough and have got it working how I want it.

What I'm trying to do is get this information into a database so I can search it using a web interface (through Rails of course).

[...]

Another idea: if this is to be read-only, you could use the XML files themselves as the DB, either through ActiveModel or (probably better) something like eXistDB.

Best,

4

Marnen Laibow-Koser wrote in post #961588:

Then you don't want to seed the database -- seeding is only for relatively static data (such as state and country names) that needs to be loaded before you ever run the application. It sounds to me like you simply want to read the XML files and build INSERT queries. And unless I'm badly mistaken, you already know how to do that.

Hi Marnen. Thank you for the quick reply. Honestly, I've never worked with xml files like this. I grabbed the code for the ruby program from a lot of different sources and kind of crammed it all together to see what would work best. I messed up a few times and destroyed some xml files in to process (thank goodness for backups!). I can output all the xml at text to the screen... But that's about the extent of my abilities. I'm not sure how to turn them into objects to insert into a database.

You may need a cron job or some such to parse new files as they come in.

Thanks for the tip! I'll look into doing that.

What's the problem, exactly? There's nothing here that you don't know.

Thanks for the confidence, but I don't have the same confidence in myself. There's a lot I don't know about Ruby and Rails, and I'm still trying to learn. I can create a database in Rails. I know how to do that. But I don't know how to get this text from the xml files turned into objects to send to the database. Any help on how to do that would be awesome.

Thanks again Marnen!

--Matt Royer

5

Solr is another option, this will make it searchable and you will be able to retrieve the source xml.

It's not really an XML database but if your data is essentially not relational then it will do a great job. You could of course put the relational aspects in a database but keep the body of the data in solr.

6

Marnen Laibow-Koser wrote in post #961590:

Another idea: if this is to be read-only, you could use the XML files themselves as the DB, either through ActiveModel or (probably better) something like eXistDB.

That would be awesome! Any links on where to go to figure out how to do that? I'll look into eXistDB.

Thanks again Marnen!

7

Peter Hickman wrote in post #961594:

Solr is another option, this will make it searchable and you will be able to retrieve the source xml.

It's not really an XML database but if your data is essentially not relational then it will do a great job. You could of course put the relational aspects in a database but keep the body of the data in solr.

Sweet! That's what I need. I need it searchable. This is for work. It's nothing time-sensitive, so I have some time to work on it. But I figured I would get cracking on some type of solution.

All of the ideas so far have been great. I'll post back with any more issues, and if people could keep the suggestions flowing, I would be forever grateful!

Thanks,

--Matt Royer

8

Matt Royer wrote in post #961592: [...]

What's the problem, exactly? There's nothing here that you don't know.

Thanks for the confidence, but I don't have the same confidence in myself. There's a lot I don't know about Ruby and Rails, and I'm still trying to learn. I can create a database in Rails. I know how to do that. But I don't know how to get this text from the xml files turned into objects to send to the database. Any help on how to do that would be awesome.

What do you really need help with here? You know how to parse the XML files to retrieve the data. You know how to insert records into a DB. Now you just need to connect the two.

Stop convincing yourself you're missing something, write some tests, and start coding. :slight_smile:

Thanks again Marnen!

--Matt Royer

Best,