We've got a healthy debate going on in the office this week. We're creating a Db to store proxy information, for the most part we have the schema worked out except for how we should store IPs. One camp wants to use 4 smallints, one for each octet and the other wants to use a 1 big int,INET_ATON.
These tables are going to be huge so performance is key. I am in middle here as I normally use MS SQL and 4 small ints in my world. I don't have enough experience with this type of volume storing IPs.
We'll be using perl and python scripts to access the database to further normalize the data into several other tables for top talkers, interesting traffic etc.
I am sure there are some here in the community that have done something simular to what we are doing and I am interested in hearing about their experiences and which route is best, 1 big int, or 4 small ints for IP addresses.
EDIT - One of our concerns is space, this database is going to be huge like in 500,000,000 records a day. So we are trying to weigh the space issue along with the performance issue.
EDIT 2 Some of the conversation has turned over to the volume of data we are going to store...that's not my question. The question is which is the preferable way to store an IP address and why. Like I've said in my comments, we work for a large fortune 50 company. Our log files contain usage data from our users. This data in turn will be used within a security context to drive some metrics and to drive several security tools.
I would suggest looking at what type of queries you will be running to decide which format you adopt.
Only if you need to pull out or compare individual octets would you have to consider splitting them up into separate fields.
If you are only going to support IPv4 addresses then your datatype in MySQL can be an
UNSIGNED INT which only uses 4 bytes of storage.
To store the individual octets you would only need to use
UNSIGNED TINYINT datatypes, not
SMALLINTS, which would use up 1 byte each of storage.
Both methods would use similar storage with perhaps slightly more for separate fields for some overhead.
Using a single field will yield much better performance, it's a single comparison instead of 4. You mentioned that you will only run queries against the whole IP address, so there should be no need to keep the octets separate. Using the
INET_* functions of MySQL will do the conversion between the text and integer representations once for the comparison.
8 bytes in
IPv4 addresses, an
UNSINGED INT is enough, which I think is what you shoud use.
I can't imagine a scenario where
4 octets would gain more performance than a single
INT, and the latter is much more convenient.
Also note that if you are going to issue queries like this:
SELECT * FROM ips WHERE ? BETWEEN start_ip AND end_ip
end_ip are columns in your table, the performance will be poor.
These queries are used to find out if a given
IP is within a subnet range (usually to ban it).
To make these queries efficient, you should store the whole range as a
LineString object with a
SPATIAL index on it, and query like this:
SELECT * FROM ips WHERE MBRContains(?, ip_range)
See this entry in my blog for more detail on how to do it: