| Vote PA HOME |
|
Sequoia Advantage Recertification Exam October 11, 2006 -- At long last, just a month before the election and after at least three postponements, Sequoia came in for its recertification test last week in Harrisburg for the Advantage AVC vote machine and - especially - its Win EDS software. Win EDS, the ballot installation and vote tabulation software, had flunked the test last March and wasn't used to tabulate the votes for the May primary. This time around the examination went much more smoothly for Sequoia, although there were still some problems. Marybeth Kuznik and I attended, wearing our white hats. All in all, though, examiner Michael Shamos, while not issuing a recommendation on-the-spot, seemed pleased and praised Sequoia for the enormous effort that he said went into making the software improvements. More tellingly, perhaps, was the cheery disposition of the four Sequoia staffers, including VP Paul Nulty, upon conclusion of the exam. Despite some pretty anxious moments, the Sequoia people and the Secretary of State's office staffers walked out happy. Shamos estimates his official report will take about two weeks to issue, but Montgomery County has already started upgrading the machines, downloading the software and starting on the Logic and Accuracy tests. Two big questions loom: 1) How significant are the problems that did arise (see below) and when and how will they be addressed? 2) What about the questions that Shamos didn't ask? On the latter, election integrity activists know very well that experts have been able to demonstrate how to hack into some of the same vote machines that Shamos has approved for PA. Then when these security threats surface based on vulnerabilities that Shamos never tested for, he professes shock and grave concern. Or he has the SoS issue some directive with apparent little effect. This report - highly non-technical - is in the interest of keeping the process as open as possible and helping others who are taking on such questions. Another issue, before turning to the problems that arose on Wednesday, is the defects that Sequoia admits they found before making the improvements. The most significant that they mentioned seemed to be their admission of long-standing problems related to straight party voting. That is, when a voter chose a straight party vote for all offices and then wrongly went back and also chose a specific candidate from the other party - inconsistent with his straight party choice - the entire straight party vote was nullified. Yet there was no way for the voter to know that. That's because the light that went on over the straight party choice when the voter selected it never went off! According to Sequoia, this is a defect that may well have existed since the machine was first introduced or at least back to 1990. Other problems Sequoia says they fixed that were never detected in certification exams involved the write-in option switch and a "boot light" turning on before the machine would allow the voter to start. What is disturbing about these problems, especially the one involving the straight party vote, is that the Montgomery County Election Board apparently wasn't any more interested in uncovering them over the life of these machines than was Sequoia. Montco has never done an undervote analysis for their machines or any other type of performance evaluation of them, as far as is known, and has no interest in doing so. This is despite the latest research that says that full-face DRE systems are plagued by "an unacceptably high residual vote rate." (Brennan Center, The Machinery of Democracy, 2006) After a promise at a public meeting prior to the primary that they would do a report on the overall functioning of the May election, the Montgomery County Department of Voter Services issued almost 5 months later a 12-line table indicating phone calls received by election district and time-of-day with three or four-word problem identifications next to each. That was it for the research and performance evaluation for a countywide election. One issue raised by Shamos at the Wednesday exam was how to ensure that the firmware on any machine corresponds to what was actually tested and approved by the ITA. There was talk of a tamper-proof seal, which may be required by the feds in the future, but, basically, Sequoia was stumped. Shamos was also "astounded" at the level of difficulty in determining what files are actually in a machine and actually dumping out what shouldn't be there. A problem that took considerable time was the inability to successfully insert ballot installation software. It took three different cartridges and a number of unusual error messages to load the software and there was talk of communication errors between the operating system and the path find for removable devices. At one point Shamos asked, "If I haven't worked for Sequoia for 12 years (referring to one of the Sequoia staffers), how would I know what to do?" Sequoia was supposed to come up with an explanation for the problem later on, but we never heard it. When he changed vote tallies by manipulating WinEDS, Shamos noted that a "tally validation" window, a new function, showed that numbers had, indeed, been changed. However, its presentation was so innocuous that it seems fair to say that unless you were looking for that precise information, it could be easily overlooked. There was not even a warning, red alert or anything that made it clear that the security of the system had been breached. A siren was suggested, jokingly. After his manipulations, Shamos went back to see if he could "cover his tracks" by erasing evidence on the activity log. In fact, there were no tracks to cover - no evidence of any tampering on the log. Shamos suggested that a mechanism to detect such acts should be applied to the event log. You think? Overall, Shamos seemed confident that there was adequate security to foil tampering with the tabulations of the kind that he tested for last March. Unfortunately, I can't say how he arrived at that conclusion. He went quickly and I may have missed it. One of the most significant issues raised - just discussed rather than exhibited - was the question of the security of the Local Area Network used by Montgomery County. Shamos briefly described a threat scenario: It's hectic on election night. Someone gets a copy of WinEDS and does manual entry of votes and sends it to the data base server. When the cartridge is inserted, it can't be read, but it is assumed that the cartridge was read earlier so it is overlooked. What's to prevent this? How is the problem uncovered? Shamos asked Director of Voter Services Joe Passarella to have the County IT Director call him to talk about what's on the network and what the protocols are. Question: how can this "test" be interpreted if it is discussed only in a private phone call? Also, anyone know if this question was ever asked in other exams? One question that I raised in the brief Q&A at the end was about the keys for getting into the machines. I pointed to the problem with the Diebold machines and asked in a general way how secure the locks are in back of the machines. Shamos responded with a round-about explanation involving Houdini's escapades (no kidding), something about since he was all tied up, you know that that couldn't have had any impact on his ability to escape. Entertaining, but what it came down to, as far as I could tell, was that, heck, we're not really putting the trust of our democratic franchise in crumbly little keys that are probably available all over the place. Not exactly persuasive or reassuring, but probably in keeping with Shamos's stated conviction that the real threats to the security of the electronic vote stem from insiders, especially people with authority and access. I wonder what Voter Services Director Passarella thought about that comment! |
A Good Show, But Questions Remain:
Sequoia AVC Advantage Recertification Exam by PA Office of Secretary
of State
Dressed in a colorful orange polka-dot tie and orange and white striped shirt, Shamos strolled about the exam room, posturing frequently in front of a large LCD projection of the screen under scrutiny newly added for those in attendance to observe. At one point he even rendered a mime of a bound Harry Houdini escaping from chains to make his point that locks on a voting machine do not equate with security. The effect was not so much of a thorough scientific examination, but rather that of a polished performance designed to show an audience how thoroughly the testing was being done. After a rather rough beginning for Sequoia with cartridges early-on failing several times to correctly load the software, the updated WinEDS program did appear to display some improvement although questions remained. One glaring problem was that event logs could be altered to remove evidence that file changes had been made or tampering had been done. Another problem showed up when an audit trail printed out from two different memory sources in the system looked very different each time it was printed. "We know that logically it is equivalent, but one gets nervous," said Shamos. As usual, the exam consisted basically of checking the system against seventeen points required by Pennsylvania law, and a scripted vote of twelve ballots. Dr. Shamos moved through the tests in about four hours and fifty-five minutes of actual exam time. During that period he tested the basic functions of the software that failed in March as well as several new features. Performing his 113th examination of a voting system, Shamos was not shy about expressing his displeasure at various characteristics displayed by the Sequoia materials. He repeatedly took the Sequoia equipment, software, and in some instances personnel to task. "If I haven't worked for Sequoia for 12 years, how would I know what to do, " Shamos pondered when a communication problem between hardware and software manifested itself. And discovering later that a screen designed to display a list of improperly altered files instead implied that they were normal, he snapped that "flashing red and a few sirens going off would certainly be nice here." But despite the numerous difficulties that occurred during this retest, when citizen observers asked what assurance the re-exam gave that problems similar to those discovered in March would not crop up later, Shamos appeared totally confident that WinEDS was now "healthy". "We have tested every function on every menu of the system," he said. "Everything worked today." Notwithstanding Shamos' confidence, this observer left with a conflicting feeling that she had just seen a fine performance by a skilled showman, but many questions still remain unanswered about the stability and security of WinEDS and the Sequoia AVC Advantage.
|
October 11, 2006 -- Entertaining showmanship
was the order of the day as Pennsylvania voting system examiner
Dr. Michael I. Shamos cast his eye over the Sequoia AVC Advantage
and its WinEDS ballot installation and tabulation software in
Harrisburg last week The testing was being redone after the WinEDS
software failed so badly in March, 2006 on the AVC Advantage
machine that the examination then had to be suspended in the
middle. Dr. Shamos noted that Sequoia had advanced the software
approximately 60 revision numbers since March, and had added
features as well as claiming to have corrected problems.