Sign in to follow this  
MadCast Maday

MadCast Asset Status updates

Recommended Posts

Teamspeak box:

 

DDoS activity caused brief packet loss
Aug 06 2016 03:29:35 PM PT

 

Between 4:40pm and 5pm CDT, we saw several attacks against customers in Chicago that caused brief periods of packet loss (less than 30 seconds each time) because our internal and external links were saturated.

As always, the targets were immediately null-routed in order to limit the damage, and we are in the process of blocking the bots involved in the attacks network-wide, to help avoid further internal saturation. In terms of external saturation (our 100+ Gbps of capacity to upstreams), we are always exploring further upgrades and have one pending right now for our Tata circuit.

Share this post


Link to post
Share on other sites

Teamspeak box:

 

Location down
Aug 08 2016 02:33:53 PM PT

 

Our primary aggregation switch in Chicago appears to have crashed, causing the location to go offline. We are currently investigating.

We already have a new switch on order to replace this one, since it has crashed twice in the past.

Update @4:43pm CDT: We are waiting for an on-site technician to arrive at the cabinet and physically survey it to determine what has happened. The main possibilities are that the switch has failed or that the PDU that it is plugged into has failed. If it is the latter, it should be a quick fix once the technician arrives; if the former, it may take a little longer for us to rearrange cables.

The replacement switch that we ordered is higher-end, with additional redundancy, including redundant PSUs.

Update @4:57pm CDT: This was a PDU failure. We have moved our switches to the other PDU in the cabinet and most customers are now back online. We are continuing to troubleshoot the failed PDU to attempt to bring up any remaining offline machines.

Update @5:11pm CDT: The failing PDU came back online after we rebooted it, but still shows a generic error condition. We will be shipping another PDU to Chicago to replace it and will notify affected customers (a small subset of all customers at the location) when it is ready to be installed, as that will require shutting down all machines attached to the PDU.

We will also post a maintenance event when we have a firm date and time for the replacement of the aggregation switch. 

Share this post


Link to post
Share on other sites

Teamspeak box:

 

Comcast problems in Chicago
Aug 14 2016 12:38:07 AM PT

 

Comcast has been struggling in Chicago recently with overloaded transit/peering connections (for a couple of months, but particularly within the last two weeks). This has caused packet loss for many Comcast customers at times. We have been working hard to optimize the traffic to Comcast as much as we can in order to limit the amount of loss that is seen.

Most recently, Comcast has been particularly bad about overloading its connection to Tata (in the Tata->Cogent direction). We have completely excluded that provider from the outbound route selection now, but since Comcast is also overloading connections to other providers, this isn't a full resolution. We are continuing to work on our selection system to have it better juggle which provider is better moment-to-moment and client-to-client.

When packet loss is seen to a client, that client may observe it in the form of poorly-responding interactive sessions, slow or interrupted data transfers, skipping in game servers, or choppy audio in voice servers.

Share this post


Link to post
Share on other sites

Webhost Box:
 
DATE: Tuesday, Aug 23, 2016
 
TIME:  3:00 AM – 5:00 AM CST  
 
DESCRIPTION: A configuration issue requires network engineers to perform an update to the core configuration to ensure stability. A brief loss in connectivity of 30s-2m may be experienced due to BGP convergence. All necessary steps to minimize downtime will be taken.

During this period, all services (remote hands, installations, work visits, etc.) will continue as normal. The NOC is available to provide up-to-date status information or additional details, should you have any questions regarding the maintenance.

Share this post


Link to post
Share on other sites

Teamspeak box:

Intermittent packet loss over Telia
Aug 24 2016 10:24:13 PM PT

We have been noticing intermittent packet loss over one of our upstreams in Chicago (Telia) tonight, on the inbound from some endpoints (from New York/Ashburn, at minimum). We have contacted Telia to ask them about this. Clients whose ISP reaches us via Telia may notice skipping or other performance problems with services in Chicago; game servers in the Battlefield series may also see some player drops, because EA's back-end service (Blaze) has a provider that is currently reaching us via Telia.

Share this post


Link to post
Share on other sites

Teamspeak box:

Charter packet loss
Aug 25 2016 01:29:49 PM PT

We have received reports from Charter customers that they are seeing intermittent packet loss to our Chicago location. The data we have seen so far suggest that the problem is occurring between Charter and Voxel (Internap's peering network). We have emailed Internap to ask them to investigate, and we have also emailed Charter. 

Share this post


Link to post
Share on other sites

Teamspeak box:

Upcoming reboot on 9/8
Sep 06 2016 11:30:53 PM PT

Between 12:30am and 4am local time for your service on Thursday, September 8, we plan to gracefully shut down your VDS, upgrade the machine hosting it to an updated version of Xen/Linux, reboot the machine, and bring your VDS back online. We expect for this process to cause approximately 30-60 minutes of downtime for your service.

Because this maintenance will require a full reboot, we recommend that you configure any services which need to always be running to start automatically when the VDS is booted.

We do not anticipate problems due to this update, but to be safe, we recommend that you externally back up any critical (irreplaceable) files before the maintenance event and before any other service adjustment.

This Xen upgrade is required because of an important Xen vulnerability. We are rebooting all of our VDS-hosting machines that are running anything but the latest Xen release the same morning.

 

*The auto-start setting I had before doesn't seem to work so well anymore and I have adjusted it.  TS should be back up now*

Share this post


Link to post
Share on other sites

Teamspeak box:

Brief problems due to several DDoS attacks
Sep 18 2016 07:12:52 PM PT

Extremely large attacks against multiple servers run by a single customer in Chicago led to brief packet loss events shortly after 9pm CDT. These attacks led to immediate null-routes, but the impact was still significant because of the sheer amount of traffic that this botnet generated (well over 100 Gbps).

We are continuing to watch for further attacks and will continue to process any that we see.

Additionally, we are in the late stages of adding another 10 Gbps of upstream capacity in Chicago. We expect it to be online within the next few days, following additional testing. 

Share this post


Link to post
Share on other sites

Website Box:

Hello,

We have been notified by our datacenter about a faulty power breaker that needs to be replaced. The maintenance has been scheduled for today, October 27, 2016 at Approximately 6PM Eastern time. This maintenance will result in downtime until the replacement service has been performed. 

At approximately 6PM Eastern time, we will begin graceful shut downs of the servers to prevent any data loss or corruption. They will then begin the procedure. Upon completion, we will then turn the servers back on. If for any reason there is a delay, we will then refrain from shutting the servers down and waiting for their confirmation to keep downtime at a minimum

Window - 6PM - 8PM Eastern Time
Expected Downtime - 30Min - 1Hr

Thank you

Team Python

 

To help avoid problem I may even turn off the forums slightly before 6 to help prevent any database potential issues.

Share this post


Link to post
Share on other sites

Hi all,

We are aware that the Teamspeak server is currently offline.  The licensing expired without notification being sent to us and as a result the Teamspeak server '"expired".  While unfortunate we have taken this opportunity to patch the server while it is offline.  We've updated our license and expect the server to be available soon.  We apologize for this oversight and hope to have the issue resolved as soon as possible.

Share this post


Link to post
Share on other sites

I am aware of the ongoing issue with access to the website. We do not have a current ETA for return to expected function; however, please use https://www.madcastgaming.com to access the website pending further resolution of this matter. Thank you for your patience.

Share this post


Link to post
Share on other sites

On 2/27/2017 at 6:07 PM, MadCast: LTTelamon said:

I am aware of the ongoing issue with access to the website. We do not have a current ETA for return to expected function; however, please use https://www.madcastgaming.com to access the website pending further resolution of this matter. Thank you for your patience.

This issue has been resolved, and the website should be now working as intended. Thank you for your patience.

Share this post


Link to post
Share on other sites

Staff is aware of multiple users having connection issues to our Discord provider.

To resolve this, we have moved from the US South region to the US Central region. You may not notice a latency decrease, but we hope to solve the intermittent packet loss from user to server with this move.

If you see a change if your Discord connectivity experience on our server: please contact an an admin. 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this