Optimal attribute remapping tool

With the Apocrypha expansion, EVE pilots may now reallocate their basic attributes up to once a year.  Decided on the best choice can be tricky, so I've written a python script which, given a skill training plan, finds the base attributes which will accomplish that plan in the minimum time.  If you give it a plan that takes about a year, and you stick to that plan, then you'll be getting close the the optimal training rate you can accomplish.

For now the script is rough around the edges and completely user-unfriendly.  I plan to turn it into a web service as soon as I can, but I'm sharing it here as an early "friends and corporation" release.

Basic command-line usage:

> python everemap.py "Shamhat Arete.xml" shamhat-plan2.xml shamhat-new.xml

First argument is your character exported from EVEMon in "long XML" format, second argument is a plan exported from EVEMon, and the third argument is a filename which will be overwritten with a copy of the character with computed best attributes substituted.  You can reload this modified character into EVEMon to verify that calculations and see if you like the resulting schedule.

Running this on Shamhat, I find:

Name: Shamhat Arete

Base: [5, 6, 10, 13, 5]

Implants: [3, 3, 3, 3, 3]

Learning: [8, 7, 7, 7, 8, 4]

Effective: [17.280000000000001, 17.280000000000001 21.600000000000001, 24.84, 17.280000000000001]

length of plan = 216

time with current attributes:  341 days, 5:43

time with best attributes:  [10, 14, 5, 5, 5] 317 days, 14:05

time with worst attributes:  [5, 5, 15, 9, 5] 348 days, 21:33

writing modified character with best attributes to shamhat-new.xml

I managed to choose nearly-pessimal attributes when starting up, so I am quite excited to try remapping.  Interestingly, even for my generalist character, the best distribution is extremely biased towards intelligence and perception.  It might be wise to smooth the distribution a little bit to account for changes in focus.  (I might add a "one step less extreme" display to the tool shortly to show you how much you'd lose by being less specialized).

Feedback is most welcome.

Code follows:

"""Script for figuring out the best attributes to remap to, given
a character and an EVEMon plan.  Chooses attributes which minimize the
training time for the given plan, so ideally give it a plan for the next year
so it is perfectly representative.

First argument is filename for a character in EVEMon "long XML" format, second
argument is filename for an EVEMon XML skill plan.

Example usage:
> python everemap.py "Shamhat Arete.xml" shamhat-plan2.xml shamhat-new.xml

Takes learning skills gained during the plan into account, but assumes that
implants remain unchanged over the course of the entire plan.

TODO:
 * actual error handling and usage message
 * usability by non-hackers
 * just turn it into a web service (e.g. on AppEngine)
 * speed it up by coalescing skills with same divisor

Christopher Hendrie """

import copy
import datetime
import gzip
import xml.dom.minidom

# EVEMon Skills file.  Edit if necessary.
evemon_skills_filename = 'C:/Program Files/EVEMon/Resources/eve-skills2.xml.gz'

# --------------------------------------------------------------------
# Utilities

def parse_text_element(container, tag):
    for elem in container.getElementsByTagName(tag):
        for node in elem.childNodes:
            if node.nodeType == node.TEXT_NODE:
                return node.data
    return None

def nosec(s):
    return ':'.join(s.split(':')[:2])

# --------------------------------------------------------------------
# Attributes and attribute calculations

# Using order that they're presented for neural remapping, also implant
# slot order.
attribute_names = [
    'intelligence',
    'perception',
    'charisma',
    'willpower',
    'memory'
]
attribute_index = {
    'intelligence': 0,
    'perception': 1,
    'charisma' : 2,
    'willpower': 3,
    'memory': 4
}

def effective_attribute(base, implant, specific_learning, learning):
    """Effective attribute value."""
    return (base + specific_learning + implant) * (50 + learning) / 50

def effective_attributes(base, implants, learning):
    return [effective_attribute(base[i], learning[i], implants[i], learning[5])
               for i in range(5)]

def gen_base_attributes():
    for i in range(5, 16):
        for p in range(5, 16):
            for c in range(5, 16):
                if i + p + c + 5 + 5 > 39: break
                for w in range(5, 16):
                    m = 39 - (i + p + c + w)
                    if m > 15: continue
                    if m < 5: break
                    yield [i, p, c, w, m]

# Learning skills are maintained as a 6-element list, the first five
# corresponding to attributes and the last as the leve of the Learning
# skill.
learning_skills = {                
    'Analytical Mind': 0,
    'Clarity': 1,
    'Eidetic Memory': 4,
    'Empathy': 2,
    'Focus': 3,
    'Instant Recall': 4,
    'Iron Will': 3,
    'Logic': 0,
    'Presence': 2,
    'Spatial Awareness': 1,
    'Learning': 5
}
def update_learning(learning, skill, levels):
    """Increment learning skills in list learning."""
    if skill in learning_skills:
        learning[learning_skills[skill]] += levels


# --------------------------------------------------------------------
# Current Character information, including attributes, skills, and implants

class Character:
    """EVE pilot information."""
    def __init__(self, filename):
        """Parse EVE pilot from EVEMon long XML format."""
        self.doc = xml.dom.minidom.parse(filename)
        self.top = self.doc.getElementsByTagName('character')[0]
        self.name = self.top.getAttribute('name')
        self.attr_elem = self.top.getElementsByTagName('attributes')[0]
        self.parse_attributes(self.attr_elem)
        self.parse_implants(
            self.top.getElementsByTagName('attributeEnhancers')[0])
        self.parse_skills(self.top.getElementsByTagName('skills')[0])

    def dump(self):
        print('Name:', self.name)
        print('Base:', self.base)
        print('Implants:', self.implants)
        print('Learning:', self.learning)
        print('Effective:', effective_attributes(self.base,
                                                 self.implants,
                                                 self.learning))

    def parse_attributes(self, attr_elem):
        self.base = []
        for name in attribute_names:
            self.base.append(int(parse_text_element(attr_elem, name)))

    def parse_implants(self, implants_elem):
        self.implants = []
        for name in attribute_names:
            bonus_list = implants_elem.getElementsByTagName(name + 'Bonus')
            if bonus_list:
                self.implants.append(
                    int(parse_text_element(bonus_list[0],'augmentatorValue')))
            else:
                self.implants.append(0)

    def parse_skills(self, skills_elem):
        self.learning = [0] * 6
        for skill_elem in skills_elem.getElementsByTagName('skill'):
            skill = skill_elem.getAttribute('typeName')
            level = int(parse_text_element(skill_elem, 'level'))
            update_learning(self.learning, skill, level)

    def set_name(self, name):
        self.name = name
        self.top.setAttribute('name', self.name)

    def set_base_attributes(self, base):
        self.base = base
        for i in range(5):
            elem = self.attr_elem.getElementsByTagName(attribute_names[i])[0]
            for node in elem.childNodes:
                if node.nodeType == node.TEXT_NODE:
                    node.data = str(base[i])
                    break

    def export_char_xml(self, filename):
        with open(filename, 'w') as f:
            self.doc.writexml(f)

# --------------------------------------------------------------------
# Skills

class Skill:
    """EVE skill definition, read from EVEMon's XML database."""
    
    def __init__(self, skill_elem):
        """Constructs a Skill from an xml.dom element skill_elem."""
        self.name = skill_elem.getAttribute('n')
        self.rank = int(skill_elem.getAttribute('r'))
        self.attr1 = attribute_index[skill_elem.getAttribute('a1')]
        self.attr2 = attribute_index[skill_elem.getAttribute('a2')]

    def __str__(self):
        return self.name + " Rank (" + str(self.rank) + ") " + \
               str(self.attr1) + '/' + str(self.attr2)

# TODO handle gzipped file directly.
def parse_evemon_skills(evemon_skills_filename):
    """Parse EVE skills from EVEMon's eve-skills2.xml format.
    Returns map of Skill objects."""
    f = gzip.open(evemon_skills_filename, 'r')
    doc = xml.dom.minidom.parse(f)
    assert doc.documentElement.tagName == 'skills'
    skills = {}
    for c in doc.documentElement.getElementsByTagName("c"):
        for s in c.getElementsByTagName('s'):
            skill = Skill(s)
            skills[skill.name] = skill
    doc.unlink()
    f.close()
    return skills

# --------------------------------------------------------------------
# Plan time computation

level_skill_points = [0, 250, 1165, 6585, 37255, 210745]

skills = parse_evemon_skills(evemon_skills_filename)

def compile_plan(plan_filename, implants, learning):
    """Parses EVEMon XML plan, and compiles it into a list of pairs:
       (SP, [i, p, c, w, m, K]).
       The time in seconds to execute the resulting plan in minutes is
       the sum of (SP / (i * base intelligence + p * base perception ...
       + m * base memory + K) )."""
    # Copy learning, it will be modified in place.
    learning = learning[:]
    result = []
    doc = xml.dom.minidom.parse(plan_filename)
    top = doc.getElementsByTagName('plan')[0]
    entries_elem = top.getElementsByTagName('Entries')[0]
    for entry_elem in entries_elem.getElementsByTagName('entry'):
        skill_name = parse_text_element(entry_elem, 'SkillName')
        skill = skills[skill_name]
        level = int(parse_text_element(entry_elem, 'Level'))
        sp = level_skill_points[level] * skill.rank
        learning_rate = (50 + learning[5]) / 50
        divisor = [0, 0, 0, 0, 0, 0]
        divisor[skill.attr1] = learning_rate
        divisor[skill.attr2] = 0.5 * learning_rate
        divisor[5] = ((implants[skill.attr1] + learning[skill.attr1]) +
                      0.5 * (implants[skill.attr2] + learning[skill.attr2])) * \
                        learning_rate
        result.append((sp, divisor))
        update_learning(learning, skill.name, 1)
    return result

def plan_duration(plan, base):
    """Returns the duration of a plan as a datetime.timedelta."""
    t = 0
    for sp, divisor in plan:
        d = divisor[5]
        for i in range(5):
            d += base[i] * divisor[i]
        t += sp / d
        ## print(sp, divisor, datetime.timedelta(seconds = 60 * sp / d))
    return datetime.timedelta(seconds = t * 60)

# --------------------------------------------------------------------
# Driver 

def main(character_filename, plan_filename, output_character_filename = None):
    character = Character(character_filename)
    character.dump()
    print()
    all_base = list(gen_base_attributes())
    print()
    plan = compile_plan(plan_filename, character.implants, character.learning)
    cur_duration = plan_duration(plan, character.base)
    print('length of plan =', len(plan))
    print('time with current attributes: ', nosec(str(cur_duration)))
    ranked = []
    for base in gen_base_attributes():
        duration = plan_duration(plan, base)
        ranked.append((duration, base))
    ranked.sort()
    print('time with best attributes: ', ranked[0][1], nosec(str(ranked[0][0])))
    print('time with worst attributes: ', ranked[-1][1], nosec(str(ranked[-1][0])))
    if output_character_filename:
        print('writing modified character with best attributes to',
              output_character_filename)
        character.set_name(character.name + ' ')
        character.set_base_attributes(ranked[0][1])
        character.export_char_xml(output_character_filename)

if __name__ == "__main__":
    import sys
    assert len(sys.argv) >= 3
    character_filename = sys.argv[1]
    plan_filename = sys.argv[2]
    output_character_filename = None
    if len(sys.argv) >= 4:
        output_character_filename = sys.argv[3]
    main(character_filename, plan_filename, output_character_filename)

Wormhole Exploration, Take 2

Flush with my success at getting in and out of wormhole space, I invited some corp-mates to join me for a scuffle with the Sleepers.  I brought my probing Tristan, and joined up with a Megathron battleship (with probes) and a Exequror miner/support ship (without probes).  We found a wormhole in Odotte which lead to a fairly crowded w-system, and went through.

The battleship went off and fought some Sleepers, the Exequror mined some decent ores (unavailable in high-security space), and I probed around for more things to do.  Then it was time to leave.

Uh oh... the wormhole was gone.  In the local channel there was already some anxious discussion about what to do.  I decided to speak up and tell people that I was probing for a new exit and would let them know when I found it;  for some reason a number of people had come in without any probe launcher and so were helpless to get themselves out.  All that gained me was a lot of worried harassment as more and more time passed without me being able to find an exit.

Both I and the battleship pilot probed, and probed, and probed, for about 3 hours without any success finding an exit.  Eventually I had to get some sleep, since I was falling off my chair and making stupid mistakes (I didn't plan to stay up that late, but I didn't want to abandon my team in w-space either).  We found tons of other sites, but no way home.

About 11 hours later I logged back on, and found things "shaken up" a bit -- this time, the first site I probed was a wormhole.  However, it led only to another w-system, not out to k-space.  The Exequror pilot and I were online and had a tough time deciding whether to go explore the next system (risking separation from the Megathron), or to stay and wait.  In the end we decided to cross over and search from there, because the wormhole wasn't giving any warnings about its mass limit.  After a while searching, we decided to return and search the original system, and finally I found a second wormhole out into high-security Empire k-space.

The Exequror got out safely;  without a probe launcher, he couldn't help any more in w-space.  I'm still in w-space now, waiting for the Megathron pilot to log in so I can guide him out.  However, there's still a glitch: he's persona non grata with the Amarr empire, and the wormhole leads out into Amarr sovereign space.  I think there's a risk the Amarr navy will destroy his battleship on sight.  So we may need to get a trustworthy corp pilot to fly his battleship out for him (I can't fly battleships, so I can't do it).

This wormhole exploration business is quite the complex adventure.  I look forward to the more lucrative side of it, where we bring sufficient force to defeat a substantial number of sleepers, and salvage their components or harvest their resources.

Lessons learned:

- train up astrometrics skills, particularly astrometric pinpointing;  the deviation in readings makes it hard to narrow down hits quickly.

- be well-rested and ready for a long haul before heading to w-space

- every ship should fit a probe launcher and lots of extra probes in case they get separated.

- Sleepers are hard, even in the mildest sites of the mildest systems

- there are valuable resources here for the taking if you can deal with the unpredictability of wormholes and w-space

- now I really want a transport ship (Occator or Viator) for leaving wormhole space with loads of valuable goods

Post Script:  My friend in the battleship logged on, and we headed out into the Amarr Empire.  Immediately he was set upon by Navy battleships, calling him out as an enemy of the state.  We ran, jumping from stargate to stargate, pursued for a dozen jumps or more until we reached Caldari territory (where he is disliked but not to the point of instant violence).  Home at last!  I think I'll take a break from probing for a little while, and clean up some almost-expired missions.

Wormhole Exploration, Take 1

I've been playing EVE Online, the stellar sandbox MMO, for a little over a month now.  CCP just released the tenth expansion to the game, Apocrypha, and the headline feature is a plethora of unstable wormholes leading into unexplored regions of space.

After a few minutes admiring the new graphics (and the new graphics bugs), I decided to try a little wormhole exploration.  All I needed was a Core Probe Launcher I module, some probes, and one level in the Astrometrics skills.  As it turns out, there was such a tight supply of probe launchers that people were charging millions of ISK for them, so I manufactured a bunch of my own (and made a tidy profit myself, too, shuttling them around to trade hubs as they came off the manufacturing line, rushing to beat the inevitable crash to realistic prices).

I quickly found a wormhole leaving Vaere, and headed in with a friend (making sure to take lots of scan probes in case we needed a way out).  It turns out that it's easy to find the encounter sites where you'll fight enemy "Sleeper" NPCs, what's more difficult is to probe down wormholes leaving, and other sites with asteroids or fullerene gases.  As I scanned around in my Tristan, my friend tried engaging the sleepers in a Thorax and took a lot of damage.

After we left, I tried going back in with my mining-equipped Vexor to do a little ninja w-space mining.  I got about 5000 m^3 of hemorphite and hedbergite into a can before the sleepers showed up and I had to flee.  Back in Vaere, I switched to an Iteron III hauler (with probe launcher, of course), to see if I could go back and snatch the ore.  As it turned out, the wormhole ran out of mass and collapsed behind me, so I became trapped in w-space in a cargo hauler.  Yikes!  The sleeper ships were still at the asteroid belt, and I barely escaped, so I didn't get the ore either.

Fortunately I was able to probe down a new exit wormhole fairly quickly: one of the first signatures I tracked down was a wormhole leading back to k-space (the normal mapped universe).  Unfortunately, it led to Baratar, deep in low-security space and 38 jumps from home. I took the plunge anyway, and made it out safely -- likely because the systems were sparsely populated.

What an adventure!  I didn't come back with any sleeper loot, but I now know how to find a wormhole, get in, get lost and stuck, and get out.

74AC240 Resonant EL Inverter

Tonight, somewhat to my surprise, I put together a breadboarded low-power inverter for electroluminescent (EL) wire, using a 74AC240 inverter, LM339 comparator, and a 100 mH inductor. I was planning on just playing with series resonance a bit, but after I got the circuit working with a 100 nF capacitor I realized there was no reason not to try it with a 20' length of EL wire.

Nice sine!





I've been trying to design a flexible full-power multi-channel EL inverter, but while the parts are on order this can be a testbed for the series resonant topology. Two inverters from the AC240 play the role of a full-bridge, driving 100 mH in series with 24 nF to +/- 5V. The resonance increases the voltage across the EL wire with each swing, limited in the end by the current that one output of the AC240 can deliver. I measured 18 mA RMS, 33 V RMS, 3200 Hz, which closely matches a theoretical ideal 100mH+24nF series resonant circuit. The neat thing about the resonance is that it turns 5V*40mA = 200mW of input power into about 600 mVA of "reactive power". The power dissipation could be much lower if lower-resistance switches were used for the bridge, or if it wasn't being driven at full duty cycle right up to the resistive limits of the AC240 chip. If we assume that the AC240 is a 20mA current source, then we'd expect to dissipate at least 100mW in the steady state even if every other component is ideal, because the AC240 only stops being able to deliver more power when the full 5V is dropped across it. 5V * 20 mA = 100 mW.

The feedback loop for this circuit is a bit dodgy. To drive a series-resonant tank using a switched voltage, you need to switch the driving voltage in phase with the output current. Because the load is almost completely reactive, the current is 90 degrees out of phase with the voltage. Either you could measure the voltage, then add 90 degrees of phase shift with some kind of filter, or you can measure the current. Many sine-wave oscillators use a transformer winding to sample the current; there's no transformer here so we can't do that. Instead, I measured the voltage drop of the AC240 inverter output, comparing it to another inverter with the same input but driving a light resistive load. When the current switches sign, the comparator trips and flips the full bridge.

In this scheme, the common-mode voltage swings from near 0V to 5V as the bridge flips, but in theory the differential voltage isn't suppose to change sign during the flip. In practice, extra transitions can be a problem. The first circuit I build, using a single inverter to drive side of the bridge, worked right away. When I tried to increase the output current by ganging together three inverters for each side, the circuit became highly unstable. The solderless breadboard probably plays a part, because the inductance of the tangle of jumpers created major ringing on fast transitions. I discovered that I could occasionally get the second circuit to work by touching the feedback network with my fingertips in just the right spot, adding some leakage resistance and capacitance that just happened to stabilize it. Unstable resonant circuits apparently make great proximity detectors and touch sensors.

It might be possible to regain stability with careful compensation of the comparator and feedback network, but I think it would be a better idea to redesign the feedback circuit, either with a sense resistor to make the differential measurement reliable, or with voltage measurement and a 90-degree phase shift. An inexpensive PLL like a 74HC4046 might help, too, by overriding temporary sense glitches and ensuring that the oscillator starts up.

So here's yet another use for the ubiquitous, BEAM-beloved 74AC240 chip.

HyperPlex lives!

Digi-Key shipped my back-ordered LEDs on Thursday, and USPS delivered them mid-day on Saturday, so I was able to complete the HyperPlex prototype last night.

It works!



I had confidence in the theory, but I'm still thrilled and somewhat surprised to see it translated faithfully into concrete physical form.

HyperPlex construction album

The main purpose of this experiment was to demonstrate the viability of "hyperplexing", which is a topology for multiplexing LEDs using symmetric complementary drive. Using 16 tri-state microprocessor I/O pins and no other support hardware (no resistors, even), this board drives 64 LEDs without any pin needing to source or sink more than about 15 mA of current. At any given time, up to 8 LEDs are lit, so it uses 1/8th duty cycle multiplexing; visually and electrically, it's similar to the standard 8x8 row/column multiplexing, but it doesn't require any transistors or high current sources or sinks.

Might it be of practical use? Sure! If you want to drive more LEDs than you have pins, but you don't want to add extra driver chips or transistors, and you want more brightness than you'd get out of "weak charlieplexing", then one of the hyperplex topologies might be just the ticket. I'll post more on the math and the design space soon. Or if you want to build a dot-matrix display using through-hole techniques, and you don't want to solder any more components than you have to, then the HyperPlex board in its current form might appeal to you.

Since the 16 output lines used up nearly all the pins of the 20-pin ATtiny2313 microcontroller I'm using, I needed a demonstration program that would be interesting without any interactivity. The program also needed to fit comfortably into less than 1k, because the auto-generated display multiplexing code alone takes up half of the 2kBytes of flash memory in the AVR.

Life to the rescue! I'm a life-long (ha) fan of Conway's Game of Life, and it can be coded quite compactly. To keep things from dying out too quickly, I added an implicit constant checkerboard border around the display, so the boundaries and corners are hospitable to life cells. The life board is seeded from a 23-bit maximum-period Linear-Feedback Shift Register, and the tortoise-and-hare algorithm is used to detect cycles. In my early tests, it seems to go quite a long time between resets; at 5 minutes per life experiment, that's 80 years before a repeat. There are many more things to play with in the software, but I'm tempted to design and build a second revision of the hardware with an In-System Programming header before going too crazy with software -- it's hazardous to my fingers to keep pulling and re-inserting the chip every time I want to try new software.

Backwards again!

The prototype HyperPlex printed circuit boards arrived in the mail today from BatchPCB. This is the first PCB I've designed, and it's exciting to see it become physical reality. There's just one small problem...




Backwards again! This image is reversed, the invoice is normal but the PCB is mirror-reversed. It turns out that Eagle, the PCB layout editor I'm using, defaults to producing mirrored Gerber files for PCB production. The HyperPlex board is so regular that I didn't catch the error on previews.

Fortunately I can still assemble the prototype HyperPlex correctly, I just have to switch sides; the only minor downside is that the silkscreen (vanity text and component placement hints) will be on the top side, hidden by the LEDs.

It'll be lucky indeed if this is the only problem! This turned out to be quite a finicky design, and I expect it'll be hard to build too with everything packed so tightly. I was prepared for the small overall size of the board (1.6" square) but didn't anticipate just how small those pads and traces would end up.

I'm still waiting for the LEDs to ship (they're on back-order), but I'll probably try a test assembly later this week with a few LEDs and a spare socket I have on hand.

Fixing the Marware Squeak, Take 3

I recently switched from a PowerBook to a MacBook Pro (a little faster, a lot flakier), but I'm still using my minimalist MarWare Sportfolio Deluxe to lug it around. The MarWare case is great, but it has a signature annoyance: the shoulder strap mounting clips squeak incessantly. I've tried oil (nope), I've tried scotch tape (works until it wears out, looks ugly), but I may finally have found the cure:

Heatshrink! A short length of 3/8" black heatshrink tubing around the clip seems to have cured the squeak. I hope the tubing will last until Apple changes notebook form factors again.