The latest EVE Online trailer, Causality, was released at PAX Prime 2010. Causality is all about calculated vengeance in EVE, and how your actions in the sandbox can come back to burn you. Watch the Causality trailer and enter to win custom gaming loot in the EVE Universal Glory Sweepstakes.

Mass testing is an effort undertaken by both CCP and the EVE player base in the war on lag. Hundreds of players have participated so far in the Summer mass tests, and we're making progress as seen in the results here. CCP is still seeking your help in our continued efforts to improve EVE's performance through weekly mass testing, and we hope you'll join us in the next test today, September 2, at 20:00 UTC.

EVE Online: Tyrannis 1.0.5 has been deployed successfully and Tranquility is now accepting connections. You can review the list of fixes and changes at our patch notes page.

Forum threads are available in the EVE Information Portal for players to give feedback and alert us to any issues they may be experiencing.

EVE Online Romania are offering pilots all over the galaxy the opportunity to win one of 4 x $50 coupons to the EVE Store in their fantastic giveaway. In order to participate in this giveaway you must have an ACTIVE EVE Online account and a valid email address. For further details on how to win simply visit their site and find out how to participate.

Have you ever wanted to bring Jita to its knees? CCP is inviting you to do just that on Sunday, September 5, 2010. We've established a dedicated node for Jita, backed up by new hardware, and now we want to put it to the test. We invite all capsuleers to join us in Jita between 20:00 and 21:00 UTC this Sunday, and help us set a new record for players logged into the trade hub.

EVE-Pirate are running a competition in which you, the talented Capsuleer, can earn one of four $50 coupons to be used in the EVE Store. On top of the $50 prize, winning authors will also be given senior author membership which gives them the ability to post new entries to EVE-Pirate.com at will. For further details please see this link.

The EVE Online Store is kicking off its Fall Sale, with special prices you won't want to miss out on. If you've ever wanted a ship model, act now while they're priced at $98.99. If the EVE: Conquests Board Game is more your speed, it's now available for $45.99. Plus, select men's and women's tees are discounted at $14.99. The sale runs through September 30, 2010, and you can see the full listing of Fall Sale specials here.

The deployment of Tyrannis 1.0.5 has been postponed to Thursday, September 2, 2010, due to issues discovered during QA testing. Deployment will begin at 11:00 UTC and is expected to be completed at 12:30 UTC. Tyrannis 1.0.5 patch notes are available for review.

Our next mass test on the Singularity test server will be held on Thursday, September 2, 2010 at 20:00 UTC and is expected to last 1.5 hours. This mass test will focus on testing jumping and module lag, as well as EVE Voice.  We're seeking at least 350 pilots, though 550 or more would be ideal.  More info about the mass test can be found in this thread, and we're seeking your feedback here.

EVE has a deadly cobra strike force team alpha of extremely dedicated and proficient developers fighting the Lagmonster tooth and nail as their only mission. Through their work and the work of others, there's now, at this moment, a perfect storm within the company and we have a great number of fixes in the pipes that will knock your socks off.

Our hope is that very soon our beloved Tranquility will be able to support fleet fights of a scale that far exceeds anything you've seen before, hopefully going beyond the roof of roughly one thousand on a dedicated node.

That's tough talk, and we mean it. We will continue to go into detail in our ongoing series of dev blogs, some of which have previously been outlined in a dev blog by CCP Zulu. As soon as the optimizations are ready they will be pushed out to Tranquility individually and you will be able to gauge the difference yourself.

Character Nodes

In this blog I am going to talk about one of the optimizations we've been working on over the past few months: Character Nodes which we have already started deploying to Tranquility with phenomenal success.

The EVE Server is architected such a way that functionality is split into logical load balancing units and these units are statically assigned to a node which is a server process running on a single CPU core (since the server process is single-threaded). As long as any individual load balancing unit does not exceed the capacity of that CPU core we're fine since we can just add more nodes if the load becomes too high. However, if a single load balancing unit needs to do more work than a single CPU core can handle then we're in trouble.

A market region (Forge, Lonetrek, etc) is an example of a load balancing unit. We have multiple market regions living on a single node and currently four nodes servicing all the market regions. If the load on the market increases we can just increase the number of nodes dedicated to that task and decrease the number of markets on a given node. What this means is that when you're in Jita and browsing the market, you're not talking to the Jita node at all and don't feel any Jita lag effects (up until the point where you buy or sell something in which case the Jita inventory system gets involved).

Another example of a load balancing unit is a solar system. Typically we will have multiple, even hundreds of solar systems living on a single node. We call these types of nodes Location Nodes. Typically these solar systems have such low load that they can be mapped onto a node with a lot of other systems in the same way as the market is. However, now comes the gotcha: If a single solar system exceeds the capacity of the CPU core we have lost the ability to further balance it.

This is the problem in solar systems like Jita and in systems where fleet fights are occurring. We cannot split the solar system up into more units and spread them out and we cannot spread the work out onto multiple cores. We are effectively stuck between a rock and a hard place (stupid GIL!).

Because of these absolute constraints it's important that any work that doesn't need to be done by the location node is taken elsewhere. Therefore things like planets (in Planetary Interaction), markets, corporations and alliances are load balanced separate of the solar system you're in and if your EVE Client calls these services (such as when you are viewing your corp bulletins) then it's talking to different node than your location node. Your client is at any one time talking to half a dozen different nodes, depending on the call context.

Because of the very strict request-response model that we employ (you click a button (make a request), the server does work, you get a response back) in our business logic a lot of the game systems lend themselves very well to distributed load balancing (e.g. away from your location). However, historically the location node has been viewed as your 'primary' node for a lot of the auxiliary logic that runs on the cluster. If a particular call doesn't have a specific place to go to it will get routed to your location node. Now, that's just lazy.

Today, after years of optimizations the only true remaining bottleneck on the tranquility cluster is the solar system location and we need to shave off every single cpu cycle that we can. With this in mind we introduced the concept of the Character Node in Tyrannis and have been moving services over to this paradigm since.

Figure 1: Node configuration

Figure 1 shows what this look like. Calls that would otherwise have gone to 'Location' are now routed to 'Character', which is a set of nodes that is very easily load balanced according to number of logged-in characters. This fits nicely with the schema that is already in place. We need roughly 8 character nodes (out of the total of 204 sol nodes in the cluster) to handle the load and this is very predictable.

Now that most of these changes have been deployed, when your client makes a call for things that aren't directly related to your location, that call will typically be routed to your dedicated character node where your character happily ‘lives' along with tens of thousands of other characters.

Since the work coming from an individual client has a long way to go before it exceeds the capacity of a single CPU core and is easily predictable, we're in a good place. The client still makes the same number of calls to the cluster as before and the amount of work done on the cluster is unchanged. All that is different is that other nodes perform the work.

Changing this isn't very glorious or filled with a lot of eureka! moments. It's mostly just eating your vegetables and rearranging logic. The benefits, however, are substantial since, as it turned out, the majority of the calls that a client makes in a given period can be serviced by any node and therefore should not go to the location node.

The results

Okay, all the wall-of-text above was just to explain how the system works. Now that you've eaten your vegetables it's time for the sugar-laden dessert. Let's see the effect this had on Jita last week, comparing the performance before and after the phase #2 of these changes were deployed to Tranquility on 12 August.

Figure 2: Network traffic on the Jita node

In Figure 2 you can see the number of calls made onto the Jita node in four consecutive runs. After moving some services over to the character nodes, up to 80% of the calls were routed elsewhere freeing Jita up for important things like inventory operations and scams in local.

Figure 3: CPU utilization on the Jita node

As expected, moving services away from the Jita node had a good effect on CPU and has allowed us to scale Jita beyond the 1400 pilot limit that has capped its population for a while. Hopefully you should not be towed to another star system when you log in on Sunday evenings for a while.

Even though the metrics that we have gathered are for the Jita node, this change will have a positive effect on all loaded nodes in the cluster--with Jita and Fleet Fight nodes benefiting the most. The reason I use Jita here is that it has a very predictable load pattern whereas fleet fights are anything but. However, the same principles apply. Before this change you would be making something like 5-10 server calls to your location node to finish jumping, each one of these calls could take a long time to complete. Now you'll be making something like 4, with the rest returning very quickly.

We're hoping you'll be able to tell the difference the next time you decide to invade your nearest friendly neighbor. :-)

Also keep in mind that the other benefit of this type of offloading is that you don't "feel" the lag as much. Take Jita for example. If you're just browsing the market you don't notice that it's laggy since you're not talking to that node, you're talking to the market node. With the character node changes that we're doing much of the user experience will be improved greatly since the buttons that you're clicking end up on a lightly loaded node, even if clicking around in space is laggy. This can make a big difference in the overall playing experience.

We have been deploying the character node changes piecemeal to Tranquility throughout August and have a few more going out over the next few weeks. These should give you better performance in fleet fights as well as allow us to push Jita's concurrent player boundaries further.

This is all well and good but keep in mind that ‘lag' hasn't been ‘fixed' once and for all and it might not be in the foreseeable future, but we are pushing the boundaries of the cluster more aggressively these days than we have been for a while. Like I mentioned before, this is just one of the many things that we are working on right now. More services are being moved to character nodes this autumn and we will have plenty of more awesome fixes aimed at fleet fights coming in over the next weeks and months.

Fleet fight tests on Singularity

As stated before, it is very important for us to get as many people as possible involved when we are testing on the Singularity test server (SiSi). Please join in when the tests are advertised and help us test the performance improvements so that they can be deployed onto Tranquility as quickly as possible.

A word about fleet fight notifications

A fleet fight which happens on a node with a hundred other systems is not a good experience for anyone involved since often times the node only has 30% left of its CPU capacity when the fight starts. You can help make sure that the solar system you will be fighting in is on a dedicated node.

Use the Fleet Fight Notification form to let us know about a pending fleet fight. Sending in a petition is not the right way to report this, you must use the form. If you report the fight well in advance we can make sure that the fight is as smooth as we can make it.

Jon Bjarnason
Technical Director
EVE Online, CCP Games

CCP and Bigfoot Networks will be giving away prizes every day during September including 30 PLEX, 30 T-Shirts and two Killer^TM 2100 network cards. In order to participate you need to "Like" both the EVE Online and Bigfoot Networks Facebook pages. Two prizes will be awarded daily and winners will be announced on Facebook. You can also participate to increase your chances of winning by visiting the CCP booth at PAX and signing up for the event. So make sure you become a fan as soon as possible, and invite all your friends along.

During downtime on Monday, August 30 and Tuesday, August 31 the EVE Online websites will be offline between 11:00 to 11:30 UTC. This downtime will be used to upgrade the firmware on all websites and will affect the eveonline.com, account management, EVE API, bug reporting and customer support sections of the website and EVE Gate.

During downtime on Wednesday, August 25, 2010, new hardware was deployed on a dedicated node for Jita and is being used to test the impact of a new processor specification and low latency memory. We encourage players to show up and give us feedback on performance in this thread. 

From the lookout post of the impregnable CCP stronghold high in the Equus Mortuus mountains, we can see the world turning beneath us. As the universe slowly cools and we turn into ever older geezers, so also do the tools of our trade evolve.

As you may know, EVE has at its core the programming language known as Stackless Python.  When development of EVE was started, all we had to go by was a pocket watch, a piece of twisted wire, Stackless Python 1.5 and an LP with a band called Randver.  Stackless Python, back then, was centered around the concept of "Continuations" and it was rather tricky to use.

Later we moved to Stackless Python 2.0.  Along with the changes to Python itself, Stackless had grown the "Tasklets" and "Channels" that we have come to know and love.

We have since tried to keep abreast of developments in Python.  At various points in time we have upgraded our codebase to use Python 2.1, 2.3 and 2.5 successively.

Due to the amount of work that it entails, both the act of integration and the impact on developers, we have successfully dodged every other version.  But with each new version come benefits:

• New language features make development simpler and more effective
•  The Python library is improved in terms of features and performance
•  The language itself gains performance improvements.
•  We stay current with a language in development.

Now, after a few years of sticking with Stackless Python 2.5, we have finally taken the leap and upgraded to 2.7.  Python 2.7 was released a few weeks ago, and subsequently, Stackless Python 2.7.

When we do an upgrade like this, it is not a simple task.  There is a lot of private modifications to Python that need to be migrated.  There is a lot of C code that needs recompiling.  And there is a lot of python code that may need some minor modifications.

For this reason, we are very careful to make sure that no unintended problems show up.  We run an extensive set of regression tests to ensure that everything works as before, and a full performance benchmark suite to verify that performance does not degrade.

As an example, we found during testing  that some client entry fields started working differently.  A user would enter a percentage, say 5.55%, and have it rounded to one fractional digit as 5.5% rather than the previous value of 5.6%.  It turned out that the rounding of floating point numbers had been subtly corrected in version 2.7 and that 5.5% is indeed the correct value (because 5.55 is actually stored as 5.54999).  To make such UI sensitive rounding cases work as expected switched to using the decimal module, a toolkit for doing arithmetic in binary coded decimal.

Python 2.7 is the last of a line.  There are no plans to continue with a python 2.8 version.  All Python development is now focused on the 3.2 version of Python.  But we won't be going there any time soon.  Moving from version 2 to 3 is a much bigger leap:  There  are widespread incompatibilities on the API level and there are substantial language changes too. And at this time there appears to be  no immediate benefit to us in switching.

What effect will the upgrade have on the player?  In the short term probably none.  But in the longer term, it will make our job of providing quality services to you simpler, more enjoyable, and easier on our arthritic fingers and receding hairlines.

I've also got us a new stylus for our gramophone.  Stackless Python 2.7 and Randver will keep things moving for quite a while yet.

 - Porkbelly

Over the last week we've published a number of detailed blogs on the myriad steps we have been taking to tackle the lag issue many players have been reporting. If you've not yet done so, we encourage you to check out the following blogs:

• Need for MOAR Speed - CCP Zulu announces the Fixing Lag series
• Fostering Meaningful Human Interaction, Through Testing and Communication - CCP Tanis shares recent mass test results
• The Long Lag - CCP Warlock discusses the problems inherent with distributed real-time systems
• And I, For One, Welcome Our New Automaton Overlords - CCP Atropos unveils the new "thin client" developer tool
• Carbon and the Core Technology Group - CCP Unifex shares CCP's Carbon framework and discusses the Core Technology Group
• Character Nodes - CCP Atlas shares the work being done on character nodes
• Module Lag: Why Not All Bugfixes Are a Good Idea - CCP Veritas reveals how module lag can be problematic

Our newest Chronicle, "Burnt," has been published.

"Burnt" is a new EVE Chronicle written by CCP Abraxas. Published every other Monday, Chronicles are intended to examine the various aspects of life in EVE. The entire list is contained here, and a comment thread for this particular story may be found here.

As we're all too painfully aware, when a few hundred pod pilots get together and exchange ammunition, the dreaded "lag monster" often comes around.  There are a number of distinct phenomena that are referred to as lag.  It is important to distinguish between them, as they typically have different causes/solutions (even if the symptoms seem the same, like "my guns didn't work"). This blog is about the type called "module lag," in which modules start to respond very poorly - sometimes for minutes late.  Specifically, this blog is about the bug that causes this for repeats/deactivations, and why it's not fixed on Tranquility (TQ) yet.

At the Council of Stellar Management (CSM) summit in June, the CSM made an excellent presentation of the issues around fleet-fight lag, specifically about the issues of modules becoming ‘stuck' and the workarounds used by players.  This provided useful insight into a problem which, until recently, we'd never been able to reproduce in a development environment.  This made the problem very tangible, and gave us a great symptom to begin digging into.

Like all the best bugs, this one has layers.  We'll start at the first thing we noticed when digging at it:  the system responsible for telling the server when modules should be turned off or repeated would get minutes behind in processing when fleet fights happen while other systems remain reasonably responsive.  This system, named Dogma, handles module activation/repeat/deactivation, as well as the actual effects of those modules.  It also handles the various skill/ship modifiers you see in the game.

This sort of lopsided performance should be easily tweaked.  Tasks on the EVE servers use a time-sharing technique called cooperative multitasking which, in short, means that a task has to willingly yield execution to other tasks, otherwise it will run forever.  In this case, it would seem the part of Dogma handling module repeat and deactivation was being too nice - yielding execution too much.

Looking at the code some, a theory emerged as to why.  There was an error case that stuck out as odd - if an effect was supposed to be stopped or repeated, but the effect system itself didn't agree that it was time yet, the code would throw up its hands and give up.  If that error case gets hit, the processing loop would yield to other systems early.  A code comment was very reassuring though - this error was supposedly "rare."

The "rare" error happened 1.5 million times in the month of June, 2010 on TQ.

Above:  A re-enactment of my expression after learning this fact

So, there's the first layer.  This error happens, Dogma doesn't finish what it wanted to do (process every module event that should happen), the work load backs up and your modules start taking minutes to respond.  The typical thing to do here would be to run a test to verify this.  Unfortunately at that point the only environment we had to test on was the Singularity server mass tests.  So, off we go, putting in a setting to test to see if simply continuing past this error and finishing processing would be enough to alleviate the module repeat/deactivation problem.  There was a solid chance that it would.

Participants of the July 15th mass test: this is what I tested that made the node go horribly, horribly wrong.  Instead of clearing through the errors and charging on to victory, the effects processing loop instead choked on them, constantly hitting them and not yielding execution to anyone else until minutes later.  Interesting result.  The next mass test was three weeks away, so back to the lab I went to cook up something to test then.

And then, our savior arrived.  The Thin Client, as outlined in CCP Atropos' blog.  These lovely little beasties, once properly tamed, allowed us to get a few hundred clients doing simple behaviors in the lab.  First, there were lazors:

Then there were drones:

Then there was orbiting:

And then there was lag!  The server I was running against was positively unhappy.  But still, modules were reasonably fine - the error was not to be seen.  (Remember what I said above about there being many manifestations of lag?  Well this was a different sort, but not what we needed to reproduce the ‘stuck module' symptom)  Some fantastic sleuthing by my teammate CCP Masterplan lead us to some simple steps that could be done to induce this error.  Module delays quickly followed.

Once we have a reproduction in the lab that we can poke at, test new code on, and generally just play with, only the craziest bugs stand a chance.  This one took only a couple hours to pin down once this setup was in place.

So layer two!  The error, as you may recall from a few paragraphs up, was that Dogma was saying that the effect wasn't ready to be stopped or repeated yet.  Well, why?  The calculation is as easy as you'd think it would be - start time plus duration.  It would take a lot to miscalculate that, so what the hell?

Turns out, the errors were coming about because the same effect was in the stop/repeat manager queue twice (or more).  One of the copies would process fine, and then all of the duplicates would generate the error because the effect was already repeated by the first one that processed fine.  So, duplicates in the manager caused the error, the error caused the process loop to yield early, the process loop yielding early caused module problems.

Next up, we put together a test to see if enforcing that there be no duplicates in the manager queue fixes the error, and therefore the module problem.  Cue the thin clients, they happily pew pew themselves for a few hours while we get the code right, and the result is a positive - fixing this layer does fix the symptom.

But good engineers don't just fix a problem once, you fix it at as many layers as you can.  At this point, we've prevented duplicate effects from entering, but they shouldn't be trying to get in there in the first place.  A quick trace dropped in where the duplicate is requested pointed us right at the top layer of this bug.  The code that's responsible for making an entity (a drone, an NPC, things like that) go docile was calling the very same function that the stop/repeat manager was calling to stop its effects!  So specific timing cases, which are rare under normal circumstances but extremely common in high-load situations, were causing the effect to repeat instead of stop.  One quick tweak there and the top layer is fixed as well.

Thin client tests confirmed that fixing either layer fixes the symptom.  Cue the mass test!

The August 5th mass test had three rounds of combat in order to help us identify the impact of these bug fixes.  The first round had the fixes turned off, and the second with them turned on, no difference in style of fighting.  The fixes did exactly what they intended to do - repeating/disabling effects went from being 155 seconds behind at the height of round one to never getting more than seven seconds behind.  Success!

Well, almost.  As anyone who was at that test would tell you, the game was in bad, bad shape.  If you could get a lock and get a module to start repeating, it would repeat with gusto, but it rarely actually happened.  Locking took minutes, same with turning on or off a module.  Even warping away took a very long time.

There's two problems evident from that test.  Firstly it's clear that while relying on a bug to yield execution from this loop is a bad idea, not yielding at all isn't a good plan either.  Secondly, and more importantly, reasonable numbers of pilots can generate enough load for the effects system that it can't keep up.  That's a dangerous game, taking  more than one second to process one second worth of effects - it quickly leads to doing nothing but process effects.  That's what we started to see happen in the second phase of combat, which is scary business.

In the third phase we wanted to see where the tipping point was, so we started off with a decreased load by requesting no drones.  Everything was peachy, the system could keep up just fine.  Adding drones in doesn't add very much load, but it was enough to tip the effects system over the edge of being able to keep up.  We were able to find where that tipping point was, and that will help guide what we do for TQ in the short term.

Essentially what we have to do is re-introduce yielding in the effect processing queue, only under our control instead of at the whim of some defect.  It will be after a configurable amount of time, and I'll be present at some big fights to try to tune in on where that value should be to balance the needs of processing effects versus every other system.  Longer term, we're working on some designs that could allow the effects system to keep up under very high load, which should finally put the nail in the coffin on module response issues.

In the meantime, we've been pointing the thin clients at various hotspots of performance and tackling them.  More on that as soon as the results are out for you to enjoy!

~CCP Veritas

PS - Big shout out to the CSM for their help to date pin pointing specific issues like this one and notifying us of fights as they happen so we can be there monitoring.  Also, thanks to all the players turning out for the Singularity mass-tests.  Much love~

Till then, toodles.

Our next mass test on the Singularity test server will be held on Thursday, August 26, 2010 at 20:00 UTC and is expected to last 1.5 hours. We would like at least 300 pilots (500 is ideal). More information about the test can be found in this thread. Feedback about the test can be shared in this thread.

For those of you interested in learning more about mass tests and our most recent results, be sure to check out CCP Tanis's newest dev blog.

I like planning. I really do. I think calendars are pretty sexy and my OCD acts up when things aren’t color coded, have point values or do barrel rolls. This summer, I planned to sneak a few smaller items onto my teams backlog and positively surprise the Council of Stellar Management (CSM) when they got here. I ended up going to all these CSM summits, talking about features new and old, but never actually got to tell the CSM about some of the small things we had lined up. My plan didn’t work out, but the job got done, which is what matters.

Usually, smaller changes get bunched up and released with the coming expansion. This particular change is a little different though, as part of the code is needed for a bugfix, meaning we will be deploying it in the near future instead of this winter. The change we are introducing is going to end ghost datacore accumulation.

Currently, any character collecting research points from an agent will continue to do so, even after your account lapses due to inactivity. This is a pretty massive loophole to making substantial amounts of money and it is now being closed. When this change is deployed, characters will stop collecting research points when your account lapses into inactivity. You won’t lose the points you’ve accumulated up to that point, but simply won’t gain anymore. When you activate your account, your character will automatically begin earning RPs again. This change is long overdue and will hopefully benefit our active players.

On a sidenote, this was an issue our team picked off the CSMs list of priorities. It’s a great list that outlines both big and small concerns in the community and hopefully we can continue to address it. Summers are pretty quiet here, due to holidays and it’s a great time to go over the list and pick a few items for your team. Hopefully, we can bring you more items off the CSMs list, but till then, here’s one of them.

If anyone is interested, the CSM item is here.

The complete list is here.

Anyway, I should have more blogs coming up as we speed up for the winter release.

Till then, toodles.

Over the last few days you will have seen the blogs coming out from the boffins in our engineering teams about Mass Testing, Long Lag and Thin Clients. In some of these blogs you will have seen reference to ‘Core' teams and so I wanted to take this chance to introduce the Core Technology Group (CTG), let you know what it is we do and where we sit within the CCP development structure.

At the end of June, our CEO Hilmar Veigar Petursson referred to CCP's Core Technology Platform at his opening keynote of China GDC. For some time now, we have been internally branding the framework we use to build all of our games as "Carbon."

Giving the framework a name helps us a lot with communication internally and of course to you, the wonderful players of EVE Online. There is also the old Icelandic saying of "if you know its name, you can kill it", which basically talks about the power of knowing the names of things and how that gives you the ability to control said phenomena. As complex and multifaceted as our Carbon framework is then we certainly benefit from all the help we can get to wield it. And the custodians of Carbon are the Core Technology Group.

This video shows some of the Carbon technology in action as was presented by two CTG members at GDC this year.

So, why have Carbon and what is it really?

As CCP and EVE continue to grow, it makes sense for us to consider our existing technologies and re-use these if appropriate across our projects as part of the Carbon framework. This becomes more appealing if those technologies are proven (or, as we call it, ‘battletested') in the crucible that is a game being used by a passionate, resourceful and sometimes devious player base. Furthermore, as our new projects continue to mature, we can take the technologies developed by them and apply them to EVE. This means that the EVE development teams get some great new technologies which it hasn't had to spend any developer time creating. Having the CTG take the strain making this happen ensures the EVE developers can concentrate on developing EVE and make best use of this common technology.

It also makes sense for a central group (the CTG) to build brand new Carbon technologies which we know can be shared. This group can then deploy and support the technology across CCP to whoever needs or wants to use it.

So what is the Carbon technology framework?

Well, it is mostly things which operate at a low level in our games. The idea is that the framework will consist of all of the key technology pieces for an MMO that our game teams can pick up and use. The Trinity2 graphics engine was the first piece of Carbon technology (even if we didn't know it as Carbon back then) introduced back in late 2007 by the newly formed Core group. Since then, various parts of our technology have been ‘Carbonised' such as the graphics refresh that happened as part of Apocrypha. You can refer back to these Dev Blogs for details of things which have been made part of Carbon in the past.

StacklessIO

Core Tech - the beginning

Of course, just creating and deploying new or existing technology, even if it is ‘battletested', is only part of the story. The CTG also spends a significant portion of its time re-working, re-writing and re-engineering parts of our codebase in a continuous effort to keep it up to date as technology advances. This also allows us to prepare the codebase to accept  new technologies into the Carbon framework, allowing us to keep pushing the boundaries of what EVE Online can be.

Who is in the Core Technology Group?

Well, the CTG started, as mentioned earlier, with a few graphics programmers working under the direction of our Chief Technology Officer to produce the next generation of the EVE graphics engine. From that small beginning, we have been hiring new people into CCP in order to fill a number of teams. The CTG is a separate group within CCP, it is not part of EVE. We do not count as part of the EVE headcount and we have a separate budget and hiring plan. However, in reality we work very closely with EVE although Core does not work on game features. We provide the reprocessed minerals that  the game teams then use to, in the case of EVE, build your serious internet spaceships.

We currently have 5 teams within the CTG, although we do use carefully chosen 3^rd parties when needed. Futuremark is a good example where we are working with them on a new part of the graphics rendering engine. It must be noted that whilst the CTG teams all work on Carbon, some parts of Carbon are not maintained by the CTG. These are usually specialist areas which can be managed by the team which has that expertise. Animation is a prime example. The animation team is based in Atlanta and the work they are doing will be part of Carbon.

The 5 Core Technology Group teams consist of the following...

2 x Core Graphics Teams (12 people):

These teams are those working in the bowels of Trinity2, making it perform better, provide sexier visuals and use more up-to-date technology. The Carbon Character Technology video from earlier in this blog demonstrates a small portion of their work-in-progress. As you can see, this team has been working on the graphics technology for avatars and the environments they will inhabit. For EVE that's Incarna. These guys also develop and maintain our in-house tools system, ‘Jessica', which is used by pretty much every developer in CCP and contains the Trinity2 rendering engine. As requests for improvements, new functionality or bug fixes come in from the EVE developers, the Core Graphics guys get on the case and deliver. Jessica is also used by the EVE video team in making all of our trailers, allowing them serious time-saving shortcuts in staging assets for dramatic narrative effect.

Core EVE Graphics Team (3 people):

This team consists of three Core Graphics developers who are 100% assigned to EVE, working as part of a nine person EVE development scrum. Due to the demands of working in a cross-project, multinational company, parts of the CTG must occasionally shift its focus onto a particular project on a temporary basis. When this concentration is not on EVE (currently it is), the Core EVE Graphics team provides the link and continuity of Core involvement in the graphics side of EVE. By staying 100% laser focused, they make sure EVE continues to get the Core graphics programmer support it needs. You may have seen the recent Dev Blog by CCP Blaze about Tyrannis Performance Improvements who is on this team. These guys are instrumental  in working closely with the EVE Art team to bring you new ships, planets and other graphical wonders.

Core Infrastructure Team (4 people):

This team is really at the heart of everything we are able to put out to our customers at CCP. These guys have created a common set of Productivity software, tools and technology which allows us to build more efficiently EVE and its patches from the source code, significantly reducing the time it takes to make EVE. This team has also built the repair tool which CCP Mandrake blogged about recently. In addition, this team has been working for a long time on delivering a way for our developers to be able to stress test EVE when they are developing new features or investigating and fixing bugs. We are now rolling this out and you can read more in the Dev Blog from CCP Atropos about the Thin Clients. I believe this is a significant step forward in how we are able to build and maintain EVE, allowing us to deliver a virtual world that we are much more confident is able to operate as we intend when we have record numbers of players hammering our servers and software.

Core Cluster Team (7 people)

What is one of the cornerstones of EVE? Lots of people interacting in a single universe without sharding. Obviously, EVE has scaled to dizzying heights over the last 7+ years and we now have more players than ever immersed in New Eden. However, we know that there are problems and we know we have to be constantly fighting to help the EVE cluster scale well beyond where it is now. This important task rests on many people all across CCP, from operations and virtual world staff to the EVE software developers to the EVE game designers and beyond. A key element in the fight to improve how we scale and reduce ‘Lag' in the game is the work of the Core Cluster team. This team has the high level goal of developing our cluster technology to allow EVE to scale well into the future, even as we put more demands on it with more pilots inhabiting a more dynamic and developed EVE universe. This is no overnight fix and there have been some great discussions (as well as some painful ones) on EVE-O and other forums. Many of these threads have a good grasp, if not of the specifics, then of the general idea that we are trying to solve some of the hardest problems in a number of very complex disciplines. CCP Warlock works as the Distributed Systems Architect for CCP and the Core Cluster team and recently released an excellent blog on some of the challenges we are facing.

As recent blogs have started to describe, we are doing things right now to attack lag. We have people looking at specific bugs and issues which have been plaguing fleet fights. We are well aware that there are lag issues around large fleet fights (and some not so large ones) which are reducing your enjoyment of this part of EVE. We know because you have been telling us about your experiences regarding things like module lag, jump in lag and blackscreens to name just a few. We also know because we, as players ourselves (long time lurker, first time poster checking in after 5 years playing) experience these issues. I will leave the specifics of our progress against these problems to the actual developers working on "lag" who can get low down and dirty with some real techy blogs.

Genuine progress on these issues is being made thanks to the tools being produced by Core Infrastructure that are being used by the Core Cluster and EVE development teams. The nature of the lag problem means that it takes time to diagnose and address the problems, but we have invested significantly in getting the right tools and people to help us identify and improve the situation as fast as possible. Once fixes have been deployed to TQ and we have real evidence that they are making a positive difference to your playing experience, you will see more Dev Blogs detailing the processes and fixes.

So in terms of teams and what we work on, that is the Core Technology Group as it stands right now. You can find some more information here. We will continue to grow proportionately to CCP's development teams and the growing scope of the Carbon framework. We are actively recruiting into the CTG and if you are interested, you can apply at the usual place, the CCP jobs page.

Now that you hopefully have a bit more clarity on how CCPs Core teams are structured, it is important that you, the EVE players, know that the focus of the Core Technology Group is on supporting EVE and the experience of playing it. Through our Carbon strategy we make sure that all future product development at CCP feeds back into Eve, as they all integrate back to Carbon. We also make sure that CCP has a robust battletested core framework to win our war against the impossible.

Molea, Khanid - On 112.08.18, Azia Burgi's cemetery was attacked by combined forces from Goonswarm Federation and TEST Alliance.

This is the second time the cemetery has come under Goonswarm's crosshairs as they were also responsible for the destruction of the first cemetery, in Y110.

According to Azia Burgi, the cemetery's caretaker, the tower guarding the 'caskets' came under fire on Monday: "It was about midnight... when around 60 Goon pilots jumped into Molea and started attacking us."

Azia explained that while three pilots of her corporation were manning defences, Goonswarm's pilots used their numbers and their remote repair capability to continue their attack virtually unscathed: "each time we primaried a ship they had 5 Guardians backing it up, " she said.

Reports indicate that the starbase protecting the cemetery was a medium tower of Amarr design. The tower lacked the tough defences found in larger ones, but it had been choosen by Azia as a good compromise on durability and available burial volume. "When up against an alliance as large as Goonswarm no amount of sentry guns will stop them," she explained.

The starbase's destruction has not affected Azia's morale. According to her, the 'caskets' have mostly remained in place and she has hinted at plans for the cemetery's future: "I will not be bullied... [but] I can't really reveal the actual plans on how i plan to rebuild while the war is still active... I am still accepting corpses" she concluded.

GalNet References

Cemetery Information
Molea Cemetery Destroyed by Jihadswarm
Capsuleer Cemetery Flourishes

Are you affected by the events in this article? Do you have information regarding another event in New Eden? If so, please contact us with any information that you may have.

Want to become a news correspondent with IC? We are recruiting.

Molea, Khanid - During the past several months, the landmark cemetery created by Azia Burgi has recovered from previous violence and is now larger than ever.

In September 110, the cemetery was destroyed by pilots belonging to JihadSwarm who considered this burial method to be a desecration of mortal remains. It has taken twenty-two months for Azia Burgi to rebuild her cemetery and to surpass its former size.

According to Taak Coram, a capsuleer who regularly flies from Placid to donate corpses to the cemetery, the site keeps growing and is now noticeably bigger than it had been in the past. Taak believes that the cemetery is a grand attraction for a backwater system like Molea.

Previous IC Articles

• A Cemetery Grows
• Molea Cemetery Destroyed by Jihadswarm
• Azia Burgi to Resume Cemetery Services
• Azia Burgi's Cemetery Resumes Operations

Are you affected by the events in this article? Do you have information regarding another event in New Eden? If so, please contact us with any information that you may have.

Want to become a news correspondent with IC? We are recruiting.

Amarr Prime - Renowned composer Khitian Maritak, whose numerous canticles earned her the coveted Theology Council title "The Holy Inspired" and whose recent historical opera, "The Trials of Jiustan and Daro," won acclaim across the cluster, has announced the her latest creation, a choral titled "Sarum Symphony."

Jamyl Sarum's mysterious and explosive return to power two years ago signalled the beginning of a new era for the Amarr Empire. This symphonic piece is to be performed in the Grand Nave of the Basilica of Saint Nhyron on Mekhios by the imperial choir Voices of the Sefrim, and will be directed by Maritak herself. The composer created the choral over the course of seven exhaustive months to commemorate the event. The Khanid-born Maritak has declared the piece to be her "magnum opus."

Criticism has been levelled for the composer's choice of venue: Saint Nhyron was a notoriously outspoken opponent of the Sarum family during the last Ardishapur reign, though no ill will has existed between the houses in recent centuries. Maritak insists that her selection of the building is purely artistic, that the "Sarum Symphony" was written specifically for performance in the basilica in order to utilize the chuch's striking acoustic properties, and that the contentious history has no bearing on the choice.

The symphony's debut performance will be held on Lightbringer's Eve, a planetary holiday celebrating the first day of springtime. Invitation to the event is limited to members of the Theology Council, the Privy Council, heirs, and the Empress herself. A live broadcast will be held across the Empire by the ACN for the benefit of all Amarr citizens. Security for the event is expected to be tight in light of the recent attack by the Bloody Hands of Matar and the ongoing Sansha threat.

Khanid Prime - Khanid Innovation and Modern Finances have announced that they now own a controlling stake in the Khanid Kingdom's largest investment bank, Samarkand Financial. The move, which is the culmination of two months of negotiations, gives Modern Finances an even stronger presence in the Khanid Kingdom and secures a strong financial partner for Khanid Innovation, one of today's most powerful Khanid corporations.

Samarkand Financial has been a large player in the Khanid financial sector for over three decades now, but until recently has had little presence outside the Kingdom. Last year marked its biggest departure from that strategy when the company helped to finance the stock purchase that propelled Mens Reppola to the CEO position at Ishukone in April.  It was that purchase that brought the company to the attention of Modern Finances.

"Modern Finances has always looked to the Khanid Kingdom as a growth market," said CEO Alasunda Yeki. "This partnership with Khanid Innovation and Samarkand Financial ensures that we will have a strong presence in the Kingdom for the long term."

Khanid Innovation CEO Ganortchar Asabona said that the deal is part of a larger strategy for the company. "Khanid Innovation has been a leader in the technology sector since our founding, but our goals have always been to become a leader in innovation cluster-wise. This partnership is another step toward that goal."

The two CEOs made it clear they had no intention to upset the current leadership of Samarkand Financial, saying that they would be keeping the executive board intact. "We bought the company because of its success and we see no reason to change up a winning team," explained Yeki.