Friday, 24 October 2014
Tuesday, 21 October 2014
Just How Big Is It?
A few days ago a user on the Eve Online forums asked the question.
How big in Gb is Tranquility please?
The answer, and the other associated information is rather interesting.
The entity known as CCP DeNormalized decided to field this question starting from the 17th of October 2014 until the 20th.
Text in bold and green is mine.
--------------------------
"DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...
We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.
We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total."
So that's quite large then.
"Over the course of the last year, the market tables have grown by about 500 GB :)"
"DB is split across 2 tiers of storage, SSD (enterprise level intel disks in v7000 enclosures) for the heavily accessed stuff and 15k SAS disks for the second 'slower' tier.
I'm not that knowledgeable abuot the SAN side and how the block sizes are handed there, but as the LUN's are given to us, we setup our volumes @ 64K block size as this is the optimal allocation for SQL (64k = 1 extent = 8x8KB pages)
We do 2 full backups per day (written across 8 files for disk melting IO dumps) which take a little over 1 hour each... Hits the disks for around 400 MB/s"
"We used to run on the RamSan's but stopped about 3 years ago. They were completely overkill for our needs and at the time expanding was rather difficult.
We had 3 units, 2x 128 GB pure flashbased units and sometime later added a third 2 TB array (this wasn't flash based if I remember)."
I thought the RamSans were more recent, how time flies.
"we do different things w/ each backup - the secondary one gets shipped to a different server for reporting purposes, but you are correct, a differential would make sense in most cases here"
A full backup each time, well two of them as above.
"In this market order clean up, we noticed that there is one crazy player that has an open order for 7 years now!!!! Any order expires after 90 days unless you update it...
So this player has been modifying this one order, at least once every 90 days, for the last 7 frickin years!! haha
There are 3 items left in his sell order - I wanted to just go buy them so there wouldn't be such a big gap in ID's :)
I got over ruled on that as well
(i sent a mail to community about this guy saying he (or she?) should get some kinda medal, so we'll see what comes of that)"
"it is a sell order, that started with 20 units - 17 sold and 3 left :)
it's only like 1.6 mil per unit and not the highest in the region."
"I'm in a little more cynical :)
I also passed it onto security so they could check if it was some long forgotten about market bot sitting in someones closet praying for someone to buy the damn things so it could end itself :)"
My money is on the market bot plinking away hoping for a sale.
"the inventory system is around 250 GB - this includes multiple indexes on the same data though so we (or you) can access it faster, etc...
every item/stack per location has a unique row in the DB, for a total of around 2.2 billion rows at the moment :)"
2.2 billion!
"I've asked our SAN guys that very question ( re: expected life of enterprise SSD's) in the past :)
They say upwards of 5 year life span depending on how many writes, etc... keep in mind these are enterprise grade SSD's (with the insane prices to boot) and have extra 'stuff' to keep them alive longer :)
100% moar stuff!!! yeaaa!!!! hehe
We've yet to lose one, but I believe we're getting close to end of life and may likely replace in the coming year or so."
How big in Gb is Tranquility please?
The answer, and the other associated information is rather interesting.
The entity known as CCP DeNormalized decided to field this question starting from the 17th of October 2014 until the 20th.
Text in bold and green is mine.
--------------------------
"DB is currently around 2.8 TB, 2 weeks ago it was 3 TB...
We do routine archiving to pull old log records, etc... out to separate databases in an attempt to run as lean as possible.
We also have difference services (sso/vgs/images) in different DB's, so while the core DB itself is around 2 TB (2.8 w/ free space in files for growth), the entire stack is closer to 5-6 TB total."
So that's quite large then.
"Over the course of the last year, the market tables have grown by about 500 GB :)"
I wonder if the growth rate is a constant or is speeding up. My experience suggests the later.
"As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB.
the inventory system takes up about 250 GB (a single table w/ 2-3 indexes)
the market system takes up around 300 GB (2-3 tables and multiple indexes)
in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :))"
Chribba ribbing for the win!
"As mentioned, the core DB is 2.8 TB currently, and the image DB is around 1.3 TB.
the inventory system takes up about 250 GB (a single table w/ 2-3 indexes)
the market system takes up around 300 GB (2-3 tables and multiple indexes)
in-game mail system is around 300 GB as well (we delete most of chribba's mail just to annoy him :))"
Chribba ribbing for the win!
300Gb of in game mail, which is essentially just text files. Daunting.
"DB is split across 2 tiers of storage, SSD (enterprise level intel disks in v7000 enclosures) for the heavily accessed stuff and 15k SAS disks for the second 'slower' tier.
I'm not that knowledgeable abuot the SAN side and how the block sizes are handed there, but as the LUN's are given to us, we setup our volumes @ 64K block size as this is the optimal allocation for SQL (64k = 1 extent = 8x8KB pages)
We do 2 full backups per day (written across 8 files for disk melting IO dumps) which take a little over 1 hour each... Hits the disks for around 400 MB/s"
"We used to run on the RamSan's but stopped about 3 years ago. They were completely overkill for our needs and at the time expanding was rather difficult.
We had 3 units, 2x 128 GB pure flashbased units and sometime later added a third 2 TB array (this wasn't flash based if I remember)."
I thought the RamSans were more recent, how time flies.
"we do different things w/ each backup - the secondary one gets shipped to a different server for reporting purposes, but you are correct, a differential would make sense in most cases here"
A full backup each time, well two of them as above.
"In this market order clean up, we noticed that there is one crazy player that has an open order for 7 years now!!!! Any order expires after 90 days unless you update it...
So this player has been modifying this one order, at least once every 90 days, for the last 7 frickin years!! haha
There are 3 items left in his sell order - I wanted to just go buy them so there wouldn't be such a big gap in ID's :)
I got over ruled on that as well
(i sent a mail to community about this guy saying he (or she?) should get some kinda medal, so we'll see what comes of that)"
"it is a sell order, that started with 20 units - 17 sold and 3 left :)
it's only like 1.6 mil per unit and not the highest in the region."
"I'm in a little more cynical :)
I also passed it onto security so they could check if it was some long forgotten about market bot sitting in someones closet praying for someone to buy the damn things so it could end itself :)"
My money is on the market bot plinking away hoping for a sale.
"the inventory system is around 250 GB - this includes multiple indexes on the same data though so we (or you) can access it faster, etc...
every item/stack per location has a unique row in the DB, for a total of around 2.2 billion rows at the moment :)"
2.2 billion!
"I've asked our SAN guys that very question ( re: expected life of enterprise SSD's) in the past :)
They say upwards of 5 year life span depending on how many writes, etc... keep in mind these are enterprise grade SSD's (with the insane prices to boot) and have extra 'stuff' to keep them alive longer :)
100% moar stuff!!! yeaaa!!!! hehe
We've yet to lose one, but I believe we're getting close to end of life and may likely replace in the coming year or so."
Damn impressive.
"The most insane part of our DB is ram IMO - we're running 512 GB of ram, so it's able to pull a tremendous amount of data into memory and that gives us great speed.
That much RAM coupled with fast SSD's (and properly indexed data) and you can do quite a bit :)
In terms of oldest item, it's a very difficult thing to find. Up until 4 years ago or so, we were recycling ID's (32 bit) so we are really able to say beyond that.
Now we use 64 bit and can trace how old items are, but I think it's fair to say that there are items in hangars from the beginning :)"
"The most insane part of our DB is ram IMO - we're running 512 GB of ram, so it's able to pull a tremendous amount of data into memory and that gives us great speed.
That much RAM coupled with fast SSD's (and properly indexed data) and you can do quite a bit :)
In terms of oldest item, it's a very difficult thing to find. Up until 4 years ago or so, we were recycling ID's (32 bit) so we are really able to say beyond that.
Now we use 64 bit and can trace how old items are, but I think it's fair to say that there are items in hangars from the beginning :)"
Monday, 6 October 2014
Long Distance Travel Changes
Often I'm loath to call them expansions but this one will be worthy of the description as change it will certainly bring.
Since the announcement the forums have gone nuts, spewing many pages per minute of comments, most of them negative.
Personally I couldn't be happier! It will bring real change to the game, makes people rethink nearly every aspect of their activities outside of empire space.
We've had some folks, in our alliance, who almost certainly injured themselves jumping to conclusions and have abandoned null sec or even the game, maybe for a while, maybe forever.
What I'm concerned about is the negative reaction to these changes having a very negative effect on server population, and null security space becoming a ghost town.
Great for making ISK, not so great for PVP and content.
I believe the numbers quoted by CCP will be reduced a bit, if not with the first release then with the next.
The coming few weeks will be interesting to see the level of hysteria the Eve Online population will reach, and what CCP will do about that before the fateful day.
Links:
Long Distance Changes
CCP Greyscale's Forum Post about some probable limitations