During the second day of the federal trial of Ross M. Colby, who is charged with hacking Palo Alto Online and other Embarcadero Media websites on Sept. 17, 2015, Colby's defense attorney attempted to show flaws in the FBI's investigation.
But while defense attorney Vicki Young may have been on the offensive Thursday morning in the San Jose courtroom, by afternoon a picture connecting Colby to the crime began to emerge as FBI cyber-crime investigators took the witness stand.
Throughout Thursday morning, Young, in an effort to create the impression that the investigation was full of holes, followed a line of questioning to assert that FBI agents did not instruct Embarcadero Media staff on how to retain evidence; that the FBI failed to secure and completely investigate evidence that might have been on the company's computers and servers; and that the agency did not "mirror" all of the data on the devices, a process in which an exact duplicate of the data would have been made.
As she had during cross-examination on Tuesday, Young tried to cast doubt that her client had accessed a crucial document containing passwords that were the keys to the company's IT infrastructure. Frank Bravo, the information technology director of Embarcadero Media, had testified that he and his staff discovered someone had gained access to an IT department Google document called "Things to Remember." The password-protected document listed usernames, passwords and email addresses used internally to access the company's corporate-level IT infrastructure settings and router.
Young asked whether the IT team ever definitively identified that the Things to Remember document had actually been hacked. She similarly questioned Chris Planessi, senior web developer, and Cesar Torres Gallegos, a former IT help desk employee, who were Thursday morning's second and third witnesses.
"Since it was so important, did anyone make any attempt to find out if the document had been accessed that night?" Young asked Bravo.
Bravo said he did not.
"You did not -- you did not make that effort -- correct?"
"Even if it was accessed, if it was not changed, I have no way of knowing. Only Google can tell us if the document had been accessed. I suspect that Google can tell if it was accessed," he said, adding that it was not his job to investigate whether it had been accessed.
Planessi said only he, Bravo and Torres Gallegos had a password to the Things to Remember document. But "anyone with the IT staff password could log into it," he said.
Planessi also revealed that someone had accessed the server as an File to Transfer client, which is used to connect one server to another. Embarcadero's IT staff does not use this type of remote connection, however, he said. He has also never used a Private Internet Access account, which allows the user to remain anonymous -- a key fact that tied into an FBI agent's testimony Thursday afternoon.
In his testimony, Bravo said that he contacted web-hosting company GoDaddy about the changes someone had made to the domain addresses. He learned that changes had been made from his username and password. He said he asked GoDaddy to preserve log files on its servers, which record information such as the time an account was used.
Young questioned Bravo about whether the FBI had done any forensic investigation of his personal and work computers, which he had used to try to contain and investigate the breach.
Bravo and Planessi said they offered the FBI whatever they were able to preserve, but they did not receive instruction from either the FBI, Palo Alto police or the Mountain View police department's cyber-crime unit, which also became involved in the case, on how to preserve the data evidence, nor about what evidence to preserve.
Young continued the same line of questioning when Embarcadero Media President and Publisher William S. Johnson took the stand. Johnson said he gave his staff permission to provide the FBI with anything they needed, he said. The FBI did talk to him about preserving logs from the servers, he said.
Scott Hellman, FBI supervising special agent in the cyber division, said early on it was important to preserve the logs to find out what IP addresses -- the string of numbers identifying specific internet connections -- had accessed the servers and email accounts of Bravo, Planessi and Torres Gallegos. Several IP addresses jumped out because they were much longer than ones usually used by Embarcadero IT staff, he said.
Hellman followed up by identifying which providers owned the IP addresses by searching through WhoIs internet lookup. When they discovered that Comcast was the provider of five of the suspicious IP addresses, investigators sent a subpoena to the company to learn the name and address of the individual who had paid to use the IP address, he said.
"We learned that the IP addresses to Comcast were used by John Colby," he said, referring to Colby's father. John Colby was the subscriber, and the IP address was in Phillipston, Massachusetts, Hellman said. He also learned that Ross Colby was associated with the Massachusetts address and that Ross Colby had an address in San Francisco, which placed him close to Embarcadero Media, which is in Palo Alto, he said.
Hellman also identified additional IP addresses linked to John Colby and one directly tied to Ross Colby. Furthermore, the IP address of the Flying Pig Bistro, which is located 150 feet from Ross Colby's apartment, was also used to access Embarcadero Media servers and email accounts.
Young questioned why the FBI didn't call in its Computer Analysis Response Team (CART), nor the Regional Computer Forensics Lab located in Menlo Park, which are trained to mirror data from computers and could have done so to Embarcadero's servers and IT computers. Hellman, who left the investigation before it was completed to transfer to Washington, D.C., said he couldn't recall if either team visited Embarcadero Media.
Young noted that, though the FBI obtained warrants and seized computers at Colby's residence, no warrants were obtained for Embarcadero computers.
Although the FBI didn't appear to have mirrored the data on Bravo's personal and work computer or from the servers, Hellman did not give much weight to obtaining forensic evidence from Embarcadero's computers. If Colby had used Remote Desktop Protocol which allows one computer to remotely access another computer, then the IP addresses would be on both computers. Getting the logs from the external computer would show what it accessed, he said.
FBI Special Agent Anthony (Monty) Frazier, who headed the case and is on the San Francisco Division's Cyber Criminal Squad, said he didn't ask for the Embarcadero computers because investigators early on found there had been clear access.
"We were looking specifically at email addresses first," he said.
Frazier said they found that more than 100 Private Internet Access IP addresses, which allow the user to remain anonymous, had been used to access Bravo's work email between late July and Sept. 11, 2015.
Planessi's email address was accessed through private access accounts more than 70 times. A series of logins to Planessi's email in late July and early August were from IP addresses connected with John Colby, he said.
Torres Gallegos' email account was logged into by private access accounts more than 100 times. On July 25 and 26, 2015, there were 20 login events into Torres Gallegos' email directly linked to Ross Colby's IP address. Records from Comcast, the provider, showed that Ross Colby subscribed to that IP address starting June 1, 2015, and leased it until Oct. 30, 2015, the date of the FBI's subpoena for the information from Comcast, Frazier said.
Torres Gallegos' email was also accessed from John Colby's account, Frazier said. In the early morning hours of Sept. 18, 2015, both Torres Gallegos' and Planessi's accounts were logged into through private access IP accounts.
Johnson testified the hack caused more than $32,000 in damages in terms of lost employee work time and the time it took to investigate and repair the destruction to the five company websites. The company lost additional money in ads that could not be displayed on the sites until they were restored, which was partially managed by the next day but not fully accomplished for a few more days. Johnson said he could not place a dollar figure on the amount of advertising revenue the company might have lost due to customers shying away because of the hack.
But the hack did more than financial damage. Companywide, the online sites attract 400,000 unique visitors each month, he said.
"The biggest loss was to our reputation. We rely on the public's view of our integrity and our trustworthiness," he said.
Testimony will continue on Friday morning in the San Jose federal courthouse, including that of Frazier, John Colby and Ross Colby's roommate.
Read more articles on Colby's trial: