Find method to store an IP in MySQL for better performance
Question:
We’ve got a healthy debate going on in the office this week. We’re creating a Db to store proxy information, for the most part we have the schema worked out except for how we should store IPs. One camp wants to use 4 smallints, one for each octet and the other wants to use a 1 big int,INET_ATON.
These tables are going to be huge so performance is key. I am in middle here as I normally use MS SQL and 4 small ints in my world. I don’t have enough experience with this type of volume storing IPs.
We’ll be using perl and python scripts to access the database to further normalize the data into several other tables for top talkers, interesting traffic etc.
I am sure there are some here in the community that have done something simular to what we are doing and I am interested in hearing about their experiences and which route is best, 1 big int, or 4 small ints for IP addresses.
EDIT – One of our concerns is space, this database is going to be huge like in 500,000,000 records a day. So we are trying to weigh the space issue along with the performance issue.
EDIT 2 Some of the conversation has turned over to the volume of data we are going to store…that’s not my question. The question is which is the preferable way to store an IP address and why. Like I’ve said in my comments, we work for a large fortune 50 company. Our log files contain usage data from our users. This data in turn will be used within a security context to drive some metrics and to drive several security tools.
Answers:
Use PostgreSQL, there’s a native data type for that.
More seriously, I would fall into the “one 32-bit integer” camp. An IP address only makes sense when all four octets are considered together, so there’s no reason to store the octets in separate columns in the database. Would you store a phone number using three (or more) different fields?
A BIGINT is 8 bytes in MySQL.
To store IPv4 addresses, an UNSINGED INT is enough, which I think is what you shoud use.
I can’t imagine a scenario where 4 octets would gain more performance than a single INT, and the latter is much more convenient.
Also note that if you are going to issue queries like this:
SELECT *
FROM ips
WHERE ? BETWEEN start_ip AND end_ip
, where start_ip and end_ip are columns in your table, the performance will be poor.
These queries are used to find out if a given IP is within a subnet range (usually to ban it).
To make these queries efficient, you should store the whole range as a LineString object with a SPATIAL index on it, and query like this:
SELECT *
FROM ips
WHERE MBRContains(?, ip_range)
See this entry in my blog for more detail on how to do it:
I would suggest looking at what type of queries you will be running to decide which format you adopt.
Only if you need to pull out or compare individual octets would you have to consider splitting them up into separate fields.
Otherwise, store it as a 4 byte integer. That also has the bonus of allowing you to use the MySQL built-in INET_ATON() and INET_NTOA() functions.
Performance vs. Space
Storage:
If you are only going to support IPv4 addresses then your datatype in MySQL can be an UNSIGNED INT which only uses 4 bytes of storage.
To store the individual octets you would only need to use UNSIGNED TINYINT datatypes, not SMALLINTS, which would use up 1 byte each of storage.
Both methods would use similar storage with perhaps slightly more for separate fields for some overhead.
More info:
- Numeric Type Overview
- Integer Types (Exact Value) – INTEGER, INT, SMALLINT, TINYINT, MEDIUMINT, BIGINT
Performance:
Using a single field will yield much better performance, it’s a single comparison instead of 4. You mentioned that you will only run queries against the whole IP address, so there should be no need to keep the octets separate. Using the INET_* functions of MySQL will do the conversion between the text and integer representations once for the comparison.
Having seperate fields doesn’t sound particularly sensible to me – much like splitting a zipcode into sections or a phone number.
Might be useful if you wanted specific info on the sections, but I see no real reason to not use a 32 bit int.
Efficient transformation of ip to int and int to ip (could be useful to you):
(PERL)
sub ip2dec {
my @octs = split /./,shift;
return ($octs[0] << 24) + ($octs[1] << 16) + ($octs[2] << 8) + $octs[3];
}
sub dec2ip {
my $number = shift;
my $first_oct = $number >> 24;
my $reverse_1_ = $number - ($first_oct << 24);
my $secon_oct = $reverse_1_ >> 16;
my $reverse_2_ = $reverse_1_ - ($secon_oct << 16);
my $third_oct = $reverse_2_ >> 8;
my $fourt_oct = $reverse_2_ - ($third_oct << 8);
return "$first_oct.$secon_oct.$third_oct.$fourt_oct";
}
Old thread, but for the benefit of readers, consider using ip2long. It translates ip into an integer.
Basically, you will be converting with ip2long when storing into DB then converting back with long2ip when retrieving from DB. The field type in DB will INT, so you will save space and gain better performance compared to storing ip as a string.
for both ipv4 and ipv6 compatibility, use VARBINARY(16) , ipv4’s will always be BINARY(4) and ipv6 will always be BINARY(16), so VARBINARY(16) seems like the most efficient way to support both. and to convert them from the normal readable format to binary, use INET6_ATON(‘127.0.0.1’), and to reverse that, use INET6_NTOA(binary)
We’ve got a healthy debate going on in the office this week. We’re creating a Db to store proxy information, for the most part we have the schema worked out except for how we should store IPs. One camp wants to use 4 smallints, one for each octet and the other wants to use a 1 big int,INET_ATON.
These tables are going to be huge so performance is key. I am in middle here as I normally use MS SQL and 4 small ints in my world. I don’t have enough experience with this type of volume storing IPs.
We’ll be using perl and python scripts to access the database to further normalize the data into several other tables for top talkers, interesting traffic etc.
I am sure there are some here in the community that have done something simular to what we are doing and I am interested in hearing about their experiences and which route is best, 1 big int, or 4 small ints for IP addresses.
EDIT – One of our concerns is space, this database is going to be huge like in 500,000,000 records a day. So we are trying to weigh the space issue along with the performance issue.
EDIT 2 Some of the conversation has turned over to the volume of data we are going to store…that’s not my question. The question is which is the preferable way to store an IP address and why. Like I’ve said in my comments, we work for a large fortune 50 company. Our log files contain usage data from our users. This data in turn will be used within a security context to drive some metrics and to drive several security tools.
Use PostgreSQL, there’s a native data type for that.
More seriously, I would fall into the “one 32-bit integer” camp. An IP address only makes sense when all four octets are considered together, so there’s no reason to store the octets in separate columns in the database. Would you store a phone number using three (or more) different fields?
A BIGINT is 8 bytes in MySQL.
To store IPv4 addresses, an UNSINGED INT is enough, which I think is what you shoud use.
I can’t imagine a scenario where 4 octets would gain more performance than a single INT, and the latter is much more convenient.
Also note that if you are going to issue queries like this:
SELECT *
FROM ips
WHERE ? BETWEEN start_ip AND end_ip
, where start_ip and end_ip are columns in your table, the performance will be poor.
These queries are used to find out if a given IP is within a subnet range (usually to ban it).
To make these queries efficient, you should store the whole range as a LineString object with a SPATIAL index on it, and query like this:
SELECT *
FROM ips
WHERE MBRContains(?, ip_range)
See this entry in my blog for more detail on how to do it:
I would suggest looking at what type of queries you will be running to decide which format you adopt.
Only if you need to pull out or compare individual octets would you have to consider splitting them up into separate fields.
Otherwise, store it as a 4 byte integer. That also has the bonus of allowing you to use the MySQL built-in INET_ATON() and INET_NTOA() functions.
Performance vs. Space
Storage:
If you are only going to support IPv4 addresses then your datatype in MySQL can be an UNSIGNED INT which only uses 4 bytes of storage.
To store the individual octets you would only need to use UNSIGNED TINYINT datatypes, not SMALLINTS, which would use up 1 byte each of storage.
Both methods would use similar storage with perhaps slightly more for separate fields for some overhead.
More info:
- Numeric Type Overview
- Integer Types (Exact Value) – INTEGER, INT, SMALLINT, TINYINT, MEDIUMINT, BIGINT
Performance:
Using a single field will yield much better performance, it’s a single comparison instead of 4. You mentioned that you will only run queries against the whole IP address, so there should be no need to keep the octets separate. Using the INET_* functions of MySQL will do the conversion between the text and integer representations once for the comparison.
Having seperate fields doesn’t sound particularly sensible to me – much like splitting a zipcode into sections or a phone number.
Might be useful if you wanted specific info on the sections, but I see no real reason to not use a 32 bit int.
Efficient transformation of ip to int and int to ip (could be useful to you):
(PERL)
sub ip2dec {
my @octs = split /./,shift;
return ($octs[0] << 24) + ($octs[1] << 16) + ($octs[2] << 8) + $octs[3];
}
sub dec2ip {
my $number = shift;
my $first_oct = $number >> 24;
my $reverse_1_ = $number - ($first_oct << 24);
my $secon_oct = $reverse_1_ >> 16;
my $reverse_2_ = $reverse_1_ - ($secon_oct << 16);
my $third_oct = $reverse_2_ >> 8;
my $fourt_oct = $reverse_2_ - ($third_oct << 8);
return "$first_oct.$secon_oct.$third_oct.$fourt_oct";
}
Old thread, but for the benefit of readers, consider using ip2long. It translates ip into an integer.
Basically, you will be converting with ip2long when storing into DB then converting back with long2ip when retrieving from DB. The field type in DB will INT, so you will save space and gain better performance compared to storing ip as a string.
for both ipv4 and ipv6 compatibility, use VARBINARY(16) , ipv4’s will always be BINARY(4) and ipv6 will always be BINARY(16), so VARBINARY(16) seems like the most efficient way to support both. and to convert them from the normal readable format to binary, use INET6_ATON(‘127.0.0.1’), and to reverse that, use INET6_NTOA(binary)