Currently Browsing: Spam

Google’s Official Response to this week’s Postini Spam-Filtering Problems

From: Google Enterprise Support <enterprise-support@google.com>
To: justin@wsg.net
Subject: Postini Services Incident Update
Date: Fri, 16 Oct 2009 01:17:21 -0400 (EDT)

Google Inc.
1600 Amphitheatre Parkway
Mountain View, CA  94043

Postini Incident Report
Service Disruption – October 13, 2009
Prepared for Postini Services Customers

Dear Postini Customer,

The following is the incident report for the issues with mail delivery and Administration Console access
that some Postini customers experienced on October 13, 2009. We understand that this service
disruption has affected our valued customers and their users, and we sincerely apologize for the impact.

Issue Summary
Beginning at approximately 10:25 PM PDT, Monday October 12 | 5:25 GMT, Tuesday October 13,
affected customers experienced severe mail delays and disruption. Also, during this time, affected
customers had intermittent access to the Administration Console, Message Center, and Search Console.
The root cause of the delivery problem was an unintended side effect of a filter update, compounded by
database issues that further slowed message processing.

Incoming messages may have been deferred; no messages were bounced from recipients or deleted. In
some cases, sending servers may have stopped resending messages after a deferral and returned
delivery failure notifications to senders. (Typically, servers are set up to retry sending for up to five days.)
During the incident, timely status information about the incident was not consistently available to
customers. We posted information on the Support Portal and from the @GoogleAtWork Twitter account;
however, customers often experienced problems accessing the portal due to load issues, and updates
were not included on the Postini Help forum. Also, the Postini status traffic lights intermittently showed a
“green light” instead of indicating the delivery delay. Customers calling in to report cases experienced
very long wait times.

Actions and Root Cause Analysis
At approximately 11:30 PM PDT, Monday October 12 | 6:30 GMT, Tuesday October 13, monitoring
systems detected severe mail flow issues and automatically directed mail flow to the secondary data
center. Upon receiving the error alerts, the Engineering team immediately began analyzing the issue and
initiated a series of actions to help alleviate the symptoms. Message processing continued to perform
poorly in the secondary data center.

Mail traffic was then directed across both the primary and secondary data centers to maximize processing
resources. During this time, Engineering temporarily disabled the Administration Console and other web
interfaces to reduce impact to the processing infrastructure. Engineering performed a set of extensive
diagnostics and tests and determined the cause to be the result of a combination of the following
conditions:

• A new filter update appears to have inadvertently impacted the mail processing systems.
• Unusual malformed types of messages triggered protracted scanning behavior, and its
interaction with filter update affected mail delivery.
• A power-related hardware failure with database storage servers reduced input/output rates. The
latency in database access reduced our overall processing capacity.

The combination of these conditions resulted in high failure rates for mail processing and the deferral of
new connections from sending mail servers.

To fix the database issue, Engineering worked with the hardware vendor to replace the faulty hardware
component. At 11:00 PM PDT, October 13 | 6:00 GMT, October 14, database disk input/output
throughput returned to normal.

At 12:30 AM PDT | 7:30 GMT Wednesday October 14, the filter update was revoked, and mail processing
returned to full capability. As a precautionary measure, Engineering continued to process a portion of
traffic through both the primary and secondary data centers. Mail processing was restored to the primary
data center at 1:39 AM PDT | 8:39 GMT. Although mail processing was at normal speed and capacity,
some users may have seen delayed messages continue to arrive in their inboxes. These potential delays
occur when the initial or subsequent delivery attempt is deferred and the sending server waits up to 24
hours before resending the same message.

Corrective and Preventative Actions
The Engineering and Support teams conducted an internal review and analysis, and determined the
following actions to help address the underlying causes of the issue and help prevent recurrence:

• Implement standard procedures for reverting filter updates as a mitigation measure and to help
speed time to resolution.
• Perform an in-depth analysis of the filter update to help ensure this class of error is not
propagated.
• Investigate the unusual malformed messages to quickly identify the message pattern and
thoroughly understand any impacts.
• Enable monitoring for notifications of the class of power failure that may affect the database
storage system.
• Determine whether the database storage servers can be configured to maintain the throughput
level during reduced power situations.
• To improve communications during incidents, we will:
◦ Post timely status updates to the Postini Help forum for better visibility.
◦ Accelerate the work to monitor and communicate the Postini services status on the
Apps Status Dashboard. The dashboard offers a single location for the latest service
status and options for RSS feeds. This will replace the traffic lights system and provide
more accurate and in-depth information.
◦ Moving forward, update the phone status message more quickly to inform customers
during an incident.
◦ Expand phone support capacity to handle spikes in call volume. This capacity is
expected to be available within the next several weeks.
◦ Update the maintenance pages with up-to-date information that are displayed when the
Administration Console is unavailable.

Over the next several weeks, we are committed to implementing these improvements to the Postini
message security service. We understand that system issues are inconvenient and frustrating for
customers. One of Google’s core values is to focus on the user, and we are committed to continually and
quickly improving our technology and operational processes to help prevent and respond to any service
disruptions.

We appreciate your patience and again apologize for the impact to your organization. Thank you for your
business and continued support.

Sincerely,
The Postini Services Team

Hey – I got on the News!

Sandy Family from the Sanford Financial Group –  who we know from our association with Talk 1300 – invited me to speak at a seminar about how to protect one’s self against identity theft.  The turnout was great – about 80 people came to the Holiday Inn on Wolf Road in Albany.  My last post on ID theft was written as a reference for the event.

My luck was pretty good that night.  In addition to being fortunate to be included on a panel with the Chief of Colonie Police, a high-profile attorney and a staffer from the State Attorney General’s Office – I got on the local news too.

Beth Wurtman from local NBC affiliatt WNYT asked me to taped some remarks in the hallway during the tail end of the event. I took some ribbing at the office too.  As the TV spot identified me as a “computer expert” – our design staff felt compelled to make stickers (see pic).  Everyone at the WSG offices was wearing one of these stickers when I came in the next day.

Here’s the video:

Barracuda Spam Firewall Named 2008 Editor’s Best Award Winner by Windows IT Pro Magazine

Barracuda Networks Inc. today announced that its Barracuda Spam Firewall was named as a gold medal winner of the “2008 Editor’s Best Awards” in the hardware category by Penton Media’s Windows IT Pro magazine.

WSG is a reseller of Barracuda appliances, and also deploys our own Barracuda for use in spam filtering for hosted client and internal email.

“The 2008 Editor’s Best Awards utilize our editors’ product knowledge and subject matter expertise to identify exceptional products that benefit the Windows IT Pro and SQL Server Magazine reader communities,” said Jeff Lewis, group publisher.  “Our editors selected winners based on the product’s strategic importance to the market, its competitive advantages and its value to the customer.

“Our editors are tough, in-the-trenches critics,” Lewis continued.  “They demand solid value and performance from the products they select.  Editor’s Best Award winners can feel proud that the products and services they developed have earned the respect and recognition of the Windows IT Pro and SQL Server Magazine editorial staff.”

Barracuda Web Filter Wins “Best of Connections 2007”

The Barracuda Web Filter was recently named the “Best of Connections 2007 Awards” winner by Windows IT Pro.  The magazine recognized the Barracuda Web Filter in the Windows category for its strategic importance to the market, competitive advantages and value to the customer. WSG is a reseller of Barracuda appliances, and also deploys our own Barracuda for use in spam filtering for hosted client and internal email.

This is just one of many accolades the Barracuda Web Filter and Barracuda Networks products have received this year.  These distinctions highlight Barracuda Networks success in addressing customers’ needs for powerful, yet affordable protection at the gateway.  Sharing these and other accolades during the sales process can help reinforce the benefits of and third-party support, for Barracuda Networks products.  For a complete list of recent product and company awards, please visit the awards section of the Barracuda Networks Web site.

Page 2 of 212